+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201021/B1a --output-run DEEP/20201021/B1a/science#step1/20240314T072405Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 9.2138 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.191 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.8271 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.7633 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/pipeline_orig.qgraph DEEP/20201021/B1a/science#step1/20240314T072405Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 6.3395 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 20.3080 seconds; current memory usage: 0.387 Gibyte, delta: 0.208 Gibyte, peak delta: 0.196 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.387 Gibyte (main), 0.375 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0298 seconds; current memory usage: 0.387 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.387 Gibyte (main), 0.375 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1302 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201021_B1a_science#step1_20240314T072405Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 1.0919 seconds; current memory usage: 0.387 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.387 Gibyte (main), 0.375 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1304 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 3.1769 seconds; current memory usage: 0.389 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.389 Gibyte (main), 0.375 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: 89a08eeb-c2d9-4479-a0eb-c6794e55ca8b
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 22707376207616)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x14a6fc13f590>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-14 07:25:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-14 07:25:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 07:25:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-14 07:25:37 proc_lsst.multi:170 [INFO]  [multi] job_id 14686
proc_lsst.multi INFO: [multi] job_id 14686
2024-03-14 07:25:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:37 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 14686
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 14686
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 14686
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x14a6fe763f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 31efa13366746faea989734dfe566764
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dc3860e4-1579-45f9-a024-9a194b98b005_isr_947524_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dc3860e4-1579-45f9-a024-9a194b98b005_isr_947524_31.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x14a6faa76e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x14a6fac67350 state=pending>
parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x14a6fc23d3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 230d18e67c83cb96c6b5dc5deb994075
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a0b527bb-645b-4d38-a92b-89247da6b626_isr_947526_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a0b527bb-645b-4d38-a92b-89247da6b626_isr_947526_14.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x14a6fc23d8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x14a6fc23d9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x14a6fc004310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash e153b32267663cf0b85d7bcad21e7487
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8a8cb7e3-3285-4d3a-a2ba-99a1534a361c_isr_947523_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8a8cb7e3-3285-4d3a-a2ba-99a1534a361c_isr_947523_13.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x14a6fc004fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x14a6fa914210 state=pending>
parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x14a6fa917490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 6fea76f493b609470e8f2f54b2e72f82
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2edb5942-9fd3-4102-9cb3-401e260cb7f4_isr_947524_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2edb5942-9fd3-4102-9cb3-401e260cb7f4_isr_947524_37.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x14a6fef2f350 state=pending>
parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x14a6fef2fe90 state=pending>
parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x14a6fbb8c0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash ed63ef2ed102c4604725ca232066d5c7
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2647de5a-5d12-43d2-b9d2-e973f021e183_isr_947420_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2647de5a-5d12-43d2-b9d2-e973f021e183_isr_947420_8.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x14a6fbb8c190 state=pending>
parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x14a6fc597a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x14a6fc595590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 9c9009d078f68c517a2b467204336356
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4679c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2af53ff0-f16c-46dd-88a8-8252892d092c_isr_947418_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2af53ff0-f16c-46dd-88a8-8252892d092c_isr_947418_21.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x14a6fc470810 state=pending>
parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x14a6fbc80910 state=pending>
parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x14a6fbc80110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 58a29cba9d1cae8f2990969b304f229e
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7b203241-bc8e-425d-93de-b3af02d8ed22_isr_947422_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7b203241-bc8e-425d-93de-b3af02d8ed22_isr_947422_40.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x14a6fef2edd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x14a6fb119210 state=pending>
parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x14a6facabc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash f6ca5171a391c689bf2db30f7f8c2d77
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/64d7f367-b32c-4c18-9463-77dd7da53f63_isr_947422_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/64d7f367-b32c-4c18-9463-77dd7da53f63_isr_947422_27.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x14a6fc0326d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x14a6fc345dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x14a6fc347290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 8599f0fc209f949ea2f7d4e18a222dc7
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/424d4475-9099-4c20-80c7-f95958d31f72_isr_947420_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/424d4475-9099-4c20-80c7-f95958d31f72_isr_947420_21.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x14a6fc346210 state=pending>
parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x14a6fef18450 state=pending>
parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x14a6fef19450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 6f0ff4cfa702b223a27d0128195ca740
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f91398c4-7511-4251-a6d4-08a581530dbc_isr_947523_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f91398c4-7511-4251-a6d4-08a581530dbc_isr_947523_16.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x14a6fef19910 state=pending>
parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x14a6fc032bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x14a6fef1b6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 5439168b6847fc3f4db1b36a87b09470
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2566c465-ae72-4e64-9626-3ce218d0ed86_isr_947524_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2566c465-ae72-4e64-9626-3ce218d0ed86_isr_947524_46.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x14a6fae9fa90 state=pending>
parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x14a6fab88550 state=pending>
parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x14a6fb8365d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash dc3aab7c8de6b2e3409ddcb48131a059
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/106ecfbf-126f-4712-bed2-4457add4682f_isr_947524_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/106ecfbf-126f-4712-bed2-4457add4682f_isr_947524_60.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x14a6fc595710 state=pending>
parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x14a6fc19e450 state=pending>
parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x14a6fc19eb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 4cd40a6b1cd7c2a3ccfe8473bbfe673b
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54438edd-a48b-41d7-a80c-9e665e40d219_isr_947420_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54438edd-a48b-41d7-a80c-9e665e40d219_isr_947420_7.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x14a6fa615810 state=pending>
parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x14a6fb9b89d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x14a6fb9b8450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 088f8ffda1b2936c7f4d48f73cfcb020
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a8c3f37c-cec8-408d-b35a-aa9f42eb7c59_isr_947526_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a8c3f37c-cec8-408d-b35a-aa9f42eb7c59_isr_947526_24.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x14a6fb3194d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x14a6fef1d650 state=pending>
parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x14a6fef1e890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 879602bfe3bb6938fac51e48fc48e3eb
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/efb96d0e-54cc-454e-a3f1-6e27b6621648_isr_947524_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/efb96d0e-54cc-454e-a3f1-6e27b6621648_isr_947524_28.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x14a6fef1fe50 state=pending>
parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x14a6feebe510 state=pending>
parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x14a6feebe590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 92d0c22ea88b3b617621a91709265c71
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f714ed3b-1da7-4bba-864e-03508fe9e2ea_isr_947524_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f714ed3b-1da7-4bba-864e-03508fe9e2ea_isr_947524_44.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x14a6feebcb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x14a6fab76890 state=pending>
parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x14a6fa7c1610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 65fb7f734d7b3b0f0dcc622f1e39144c
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54874989-ff39-4d0c-939f-86b577b9d889_isr_947420_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54874989-ff39-4d0c-939f-86b577b9d889_isr_947420_27.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x14a6feebfd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x14a6fc41ea10 state=pending>
parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x14a6fc4759d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash c6cf3b4862d2a989e0dcd3b5dfca5ce5
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/86490b56-258d-41dc-8d44-68a2777b1207_isr_947526_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/86490b56-258d-41dc-8d44-68a2777b1207_isr_947526_21.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x14a6fc477190 state=pending>
parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x14a6fc1cbcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x14a6fc3c0d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash f510aadef85a68f587353cb51e31874d
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dd084417-5f1c-4ff9-bb09-badcda5f8116_isr_947528_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dd084417-5f1c-4ff9-bb09-badcda5f8116_isr_947528_34.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x14a6fc3c1990 state=pending>
parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x14a6fad41310 state=pending>
parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x14a6fad42bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 97f2d1afc3e505fb1755f624732f08a7
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/436e0914-6ce5-4eec-9240-1e61a127c9a7_isr_947422_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/436e0914-6ce5-4eec-9240-1e61a127c9a7_isr_947422_33.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x14a6faf47250 state=pending>
parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x14a70b3d6fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x14a70b3d4390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 447c340cbba14e5c357ecf29bdf79c8d
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ee80558e-a70b-4e42-9277-bfe16a45eed3_isr_947524_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ee80558e-a70b-4e42-9277-bfe16a45eed3_isr_947524_36.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x14a6fef36110 state=pending>
parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x14a6fb0cdb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x14a6fb0cc950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash dab2031f935fdcb5da9790e6d77692aa
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/eea00512-d9dd-4da4-bb0c-8c24620753cc_isr_947422_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/eea00512-d9dd-4da4-bb0c-8c24620753cc_isr_947422_5.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x14a6fb0cdf10 state=pending>
parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x14a6fb799a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x14a6fc6b26d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash c2fbdacf074a2e022d40646399688353
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/6118c3d2-f2f8-425a-895d-fb06a5816e43_isr_947420_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/6118c3d2-f2f8-425a-895d-fb06a5816e43_isr_947420_58.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x14a6fb9f9010 state=pending>
parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x14a6fafc6750 state=pending>
parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x14a6feece5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash ef02268a7f9cf90a487582d767d973d2
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7e5cc74c-101d-4f6f-b453-ebf6401bc599_isr_947528_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7e5cc74c-101d-4f6f-b453-ebf6401bc599_isr_947528_26.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x14a6fb9f9c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x14a6feeccc90 state=pending>
parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x14a6feeced10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 65f58f75544c3ee8067760562e1c2c09
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a9dc563-c252-4759-8c01-923a6675a660_isr_947524_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a9dc563-c252-4759-8c01-923a6675a660_isr_947524_57.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x14a6feedd210 state=pending>
parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x14a6feedc5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x14a6fbed5ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 8fc8175d2f63a462db8989b4152a7878
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/acde34d5-141f-4576-b8d7-bc5a8c67357b_isr_947420_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/acde34d5-141f-4576-b8d7-bc5a8c67357b_isr_947420_13.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x14a6feedc810 state=pending>
parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x14a6fa97ad50 state=pending>
parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x14a6feec4690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 9501c7fada462cdec09b29306952e4c0
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/75da4b06-9599-4f47-8eef-9115c02d470d_isr_947526_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/75da4b06-9599-4f47-8eef-9115c02d470d_isr_947526_52.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x14a6feec7f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x14a6feec7d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x14a6feec5dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 705e53d1704bf3fa08e2a1d3299e9a42
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b6a5f6d7-5b56-4ce5-8d04-fdd48f6e7a0f_isr_947523_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b6a5f6d7-5b56-4ce5-8d04-fdd48f6e7a0f_isr_947523_36.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x14a6feec4a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x14a6fb2a6490 state=pending>
parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x14a6fb2a7010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 036f61cbfdd1b5fd95e37ed3df1390ce
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4b183dc3-bb24-4d99-8cab-077a420134a3_isr_947528_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4b183dc3-bb24-4d99-8cab-077a420134a3_isr_947528_61.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x14a6fb2a5110 state=pending>
parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x14a6fbfe5ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x14a6fbfe6350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 04b384bfd5067004c447ce97f7e5c108
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/60d16572-8a1c-434f-a11d-87c159dfe6fa_isr_947528_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/60d16572-8a1c-434f-a11d-87c159dfe6fa_isr_947528_19.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x14a70bbe4750 state=pending>
parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x14a6fad27d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x14a6fb7905d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 477729f1754dc8155c0f3780712d8d91
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ca052227-1f08-4f19-a9e8-d022844d8894_isr_947524_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ca052227-1f08-4f19-a9e8-d022844d8894_isr_947524_23.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x14a6fbfe4e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x14a6fb1b7d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x14a6fb1b5c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash dc5fc4558e85c206d571729c75bb227b
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/ffec0d9e-4c97-4604-89a8-e3702956512f_isr_947418_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/ffec0d9e-4c97-4604-89a8-e3702956512f_isr_947418_39.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x14a6fb1b6550 state=pending>
parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x14a6fb1ea390 state=pending>
parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x14a6fad4d6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 3a140c5a8c82950372d2f0fdcd41bf74
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1d2c75d2-b93f-44b4-9843-e8c536eb1c15_isr_947528_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1d2c75d2-b93f-44b4-9843-e8c536eb1c15_isr_947528_44.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x14a6fad4e210 state=pending>
parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x14a6fbcd2250 state=pending>
parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x14a6fcf7aad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 59c49320808107f4ef7fb2ebdb4d557a
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2d2f199c-373b-46f5-bde1-87ad00150a36_isr_947523_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2d2f199c-373b-46f5-bde1-87ad00150a36_isr_947523_57.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x14a6fcf7a250 state=pending>
parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x14a6fbda7450 state=pending>
parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x14a6fbda43d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash c7f8a5d2a5c2392bbd466007b68f0a2e
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/680cb945-a656-4ed6-b830-cc6cbac4eb3d_isr_947526_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/680cb945-a656-4ed6-b830-cc6cbac4eb3d_isr_947526_13.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x14a6fab9f010 state=pending>
parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x14a6feea6590 state=pending>
parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x14a6feea69d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash cf1fec07adda2e1464f0754e1e642e29
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e956c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/62ec4331-18f7-4097-894b-f712c92fdd3c_isr_947528_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/62ec4331-18f7-4097-894b-f712c92fdd3c_isr_947528_10.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x14a6feea7d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x14a6feefd510 state=pending>
parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x14a6feefe410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash a680b31e069a44650742147a35c4c348
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e959e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/79632602-58fe-4a13-afab-84824d85f15b_isr_947523_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/79632602-58fe-4a13-afab-84824d85f15b_isr_947523_32.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x14a6facfd9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x14a6fa9d3d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x14a6fa9d3ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 1418e9a4a86d54c11635725bbe378596
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/67e78330-8374-422d-8db5-ce9c4d1850a1_isr_947528_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/67e78330-8374-422d-8db5-ce9c4d1850a1_isr_947528_25.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x14a6fafe1d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x14a6fafe2a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x14a6fb619c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 7f8ea061340bfe0fce76de60599073fb
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5b425dc5-3050-4b25-a814-e0c6be4ea908_isr_947418_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5b425dc5-3050-4b25-a814-e0c6be4ea908_isr_947418_44.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x14a6fb8f16d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x14a6fd2d1990 state=pending>
parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x14a6fb2fed90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash cc3c1fbd2e188e777f8dbdb007ef4937
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f15bfe16-cae1-4a6d-8677-79b2e2292eb0_isr_947523_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f15bfe16-cae1-4a6d-8677-79b2e2292eb0_isr_947523_51.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x14a6faaf38d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x14a6feed0d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x14a6feed3810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 50c5aecd50d4595e241960b4110b1d9a
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fa36c989-89c7-4d73-a7e8-624f35727681_isr_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fa36c989-89c7-4d73-a7e8-624f35727681_isr_947418_2.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x14a6fbcc4e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x14a6feed2250 state=pending>
parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x14a6ff06a210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash ce0fa4146fe4b53a0c4ff6bf352b4733
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/67de567e-a45f-4f1a-9d4d-a65242858998_isr_947418_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/67de567e-a45f-4f1a-9d4d-a65242858998_isr_947418_12.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x14a6ff06a890 state=pending>
parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x14a6ff06bb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x14a6ff0698d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 5a9931a127928acaf204e48dcddd60ed
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/b8d007f2-e75a-4089-8126-11f802a05e1d_isr_947524_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/b8d007f2-e75a-4089-8126-11f802a05e1d_isr_947524_54.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x14a6ff06aed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x14a6fee9f9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x14a6faa0fb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 0f73b77a8d54989b73e5ac1a3c832be1
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8f3d1aa2-0d2f-44ec-9623-6b0466efbb10_isr_947528_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8f3d1aa2-0d2f-44ec-9623-6b0466efbb10_isr_947528_6.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x14a6faa0eb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x14a6fbb62c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x14a6fbb63150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 4d8bb5cc00cc1530284ec973aa619875
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d4e2d5a3-bd32-445d-8208-6f196a89bfca_isr_947526_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d4e2d5a3-bd32-445d-8208-6f196a89bfca_isr_947526_40.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x14a6fbb63f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x14a6fb7b9210 state=pending>
parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x14a6fb7baa90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash c396c72b459c9f3d48659af9235cbb6c
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2789f2ac-7510-4998-8bf0-323bab83b63c_isr_947528_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2789f2ac-7510-4998-8bf0-323bab83b63c_isr_947528_47.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x14a6fb71db50 state=pending>
parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x14a6fb71c4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x14a6fb708b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 59ebc80fea5cef5ac194e573a9eb2227
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/07d0f5c8-769b-4696-b0a4-f3604df46e6b_isr_947422_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/07d0f5c8-769b-4696-b0a4-f3604df46e6b_isr_947422_60.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x14a6fb71e0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x14a6fb708ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x14a6fb481b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash cf9281172da8bbe8e7bdccbd3992d288
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3ce0d6c-3d56-46a5-857f-b06309ef4dce_isr_947523_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3ce0d6c-3d56-46a5-857f-b06309ef4dce_isr_947523_59.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x14a6fb481e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x14a6fb562bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x14a6fb5610d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash ba5eb6d71ce733842e089cc0af97739b
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/33ded963-9b80-4b5d-b779-890eb0602868_isr_947523_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/33ded963-9b80-4b5d-b779-890eb0602868_isr_947523_46.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x14a6fb5633d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x14a6fb373cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x14a6fb370a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash bb79b34f5c7e4b6008969c898a17d6d9
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3537d5af-bca9-4ff7-9a21-3d43c0a823a9_isr_947418_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3537d5af-bca9-4ff7-9a21-3d43c0a823a9_isr_947418_14.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x14a6fb371a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x14a6fb240f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x14a6fb240390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 790f0de8391890fdadb50bd3a583fc67
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8ac755b3-0a1c-4a1b-8246-f2d1e8f43277_isr_947523_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8ac755b3-0a1c-4a1b-8246-f2d1e8f43277_isr_947523_21.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x14a6fa9d36d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x14a6fb0f9cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x14a6fb0f8790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 8c592f4531543c03634fc3d58ffbc71e
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c1f7bcbc-51e4-4a80-ac64-034c56468f41_isr_947526_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c1f7bcbc-51e4-4a80-ac64-034c56468f41_isr_947526_22.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x14a6fb04e510 state=pending>
parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x14a6fadec810 state=pending>
parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x14a6faed50d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash fabcd511077fdd8e4562d641c3c199c4
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/abd83f4c-a7a9-4813-bd1f-3419a82cc305_isr_947420_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/abd83f4c-a7a9-4813-bd1f-3419a82cc305_isr_947420_52.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x14a6faed7cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x14a6faed4fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x14a6fad0b410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash d0d0a7f1aeb68ae875bf1a88970a653e
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/eb2cfc6e-de34-4389-9823-bd24711f2751_isr_947418_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/eb2cfc6e-de34-4389-9823-bd24711f2751_isr_947418_4.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x14a6fad0ab10 state=pending>
parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x14a6ff055390 state=pending>
parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x14a6ff0555d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash b61eab59287ea052a7a6f7e6a41c085a
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/95365e2c-8aa5-4f35-9f62-e42306772a12_isr_947524_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/95365e2c-8aa5-4f35-9f62-e42306772a12_isr_947524_14.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x14a6ff054fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x14a6ff0884d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x14a6ff08bd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash afbda6e30937b15f34c41c3181475dea
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/e5ce27d4-a452-4b74-94a6-cd18199b2808_isr_947523_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/e5ce27d4-a452-4b74-94a6-cd18199b2808_isr_947523_6.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x14a6ff088190 state=pending>
parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x14a6fa7f5290 state=pending>
parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x14a6fa70b410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash e08f00a04f6dc86c03d1c6ca11b453f2
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/55d97697-dd9a-47ea-8649-d676347ea4a5_isr_947418_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/55d97697-dd9a-47ea-8649-d676347ea4a5_isr_947418_53.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x14a6fa6f7a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x14a6fce8a410 state=pending>
parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x14a6fce88610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 4c9edd49eb171a9f98853ebe592518cf
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/374fcd2a-b964-476c-8fa0-d4c4d98968a9_isr_947420_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/374fcd2a-b964-476c-8fa0-d4c4d98968a9_isr_947420_60.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x14a6fce88990 state=pending>
parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x14a6fce8a790 state=pending>
parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x14a6fc450cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash b646098fbb2c2badcc27a69e6f2ba39c
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/24ce32cd-7873-4754-892d-5432ceba0974_isr_947418_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/24ce32cd-7873-4754-892d-5432ceba0974_isr_947418_27.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x14a6fc4503d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x14a6ff079bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x14a6ff07a010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 8612627cfca317a9cd6aaa40837fa7f3
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0ce01b95-ee20-4d6d-9905-9e32006a78cf_isr_947418_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0ce01b95-ee20-4d6d-9905-9e32006a78cf_isr_947418_49.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x14a6ff078810 state=pending>
parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x14a70150cfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x14a70150f2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 2d71743060ce278f294eec4270977274
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/82958b06-7bd7-4184-abef-f366f4db9810_isr_947524_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/82958b06-7bd7-4184-abef-f366f4db9810_isr_947524_41.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x14a70150c090 state=pending>
parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x14a6fb729050 state=pending>
parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x14a6fb729250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash f89d32ec42c5545546d249cd8d076c7b
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ca29301-2b94-44b6-96dc-2b007a7d9ea7_isr_947524_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ca29301-2b94-44b6-96dc-2b007a7d9ea7_isr_947524_24.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x14a6fb7283d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x14a6fcf9e690 state=pending>
parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: <AppFuture at 0x14a6fcf9d0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 99c1934268d27b7ba72dbee15852d960
parsl.dataflow.memoization INFO: Task 186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f977efd3-7dba-46bf-adb5-f29057d1ae35_isr_947420_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f977efd3-7dba-46bf-adb5-f29057d1ae35_isr_947420_4.stderr
parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186
parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: <AppFuture at 0x14a6fcf9db50 state=pending>
parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187
parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: <AppFuture at 0x14a6fc2e3410 state=pending>
parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: <AppFuture at 0x14a6fc3d3f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 246d1d68a5af9ca5a10040f5b9ff1d14
parsl.dataflow.memoization INFO: Task 189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7eb989fc-9b6a-46f5-892e-f0723bf68cb0_isr_947523_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7eb989fc-9b6a-46f5-892e-f0723bf68cb0_isr_947523_10.stderr
parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189
parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: <AppFuture at 0x14a6fc3d1c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190
parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: <AppFuture at 0x14a6fc2c6890 state=pending>
parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: <AppFuture at 0x14a6fb28e050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 192 has memoization hash d6fb6a7744d11a893ce32714e03bb189
parsl.dataflow.memoization INFO: Task 192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/dc4bb19b-5c4b-4743-93a2-9d9227e52a2a_isr_947526_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/dc4bb19b-5c4b-4743-93a2-9d9227e52a2a_isr_947526_30.stderr
parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192
parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: <AppFuture at 0x14a6fb28cf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193
parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: <AppFuture at 0x14a6fb28dc90 state=pending>
parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: <AppFuture at 0x14a6fc0ec650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 471b1027bfbd111bec6e8a5ca9886873
parsl.dataflow.memoization INFO: Task 195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5ddfd005-b13f-4d45-a33d-bd1f2526da5c_isr_947420_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5ddfd005-b13f-4d45-a33d-bd1f2526da5c_isr_947420_50.stderr
parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195
parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: <AppFuture at 0x14a6fc0efa90 state=pending>
parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196
parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: <AppFuture at 0x14a6fc28a310 state=pending>
parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: <AppFuture at 0x14a6fc288ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 198 has memoization hash c73fca0fe0c224304baa4c85abbf9609
parsl.dataflow.memoization INFO: Task 198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f8bf7cd9-7f9a-4b7b-b92e-6fcab7b6a897_isr_947418_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f8bf7cd9-7f9a-4b7b-b92e-6fcab7b6a897_isr_947418_30.stderr
parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198
parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: <AppFuture at 0x14a6fc28a210 state=pending>
parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199
parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: <AppFuture at 0x14a6fc28bc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: <AppFuture at 0x14a6fc0f7590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 02144213b0063e8cd0bb5ccf8174d8dc
parsl.dataflow.memoization INFO: Task 201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f5cdf599-3dd4-4dbb-8a4b-0e98118fd6b2_isr_947422_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f5cdf599-3dd4-4dbb-8a4b-0e98118fd6b2_isr_947422_59.stderr
parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201
parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: <AppFuture at 0x14a6fc0f7190 state=pending>
parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202
parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: <AppFuture at 0x14a6fbde5190 state=pending>
parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: <AppFuture at 0x14a6fbde4550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 0cc2cc1c0a9e6a8a709ce60f5c7046ed
parsl.dataflow.memoization INFO: Task 204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8621e365-b272-4c54-8c82-9caaecbda208_isr_947523_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8621e365-b272-4c54-8c82-9caaecbda208_isr_947523_48.stderr
parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204
parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: <AppFuture at 0x14a6fbde55d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205
parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: <AppFuture at 0x14a6ff014950 state=pending>
parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: <AppFuture at 0x14a6ff0169d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 09f4e29fe35bd4e03f6b22ede7ee3acc
parsl.dataflow.memoization INFO: Task 207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75f61e7d-49a4-4817-8b3f-66d08d3ac3f4_isr_947524_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75f61e7d-49a4-4817-8b3f-66d08d3ac3f4_isr_947524_17.stderr
parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207
parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: <AppFuture at 0x14a6ff015590 state=pending>
parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208
parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: <AppFuture at 0x14a6ff016510 state=pending>
parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: <AppFuture at 0x14a6fb744e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 12531a986830e830b2ecb477ce40106d
parsl.dataflow.memoization INFO: Task 210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/7fecb5ab-fe04-47c7-a527-a4209efd2678_isr_947418_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/7fecb5ab-fe04-47c7-a527-a4209efd2678_isr_947418_41.stderr
parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210
parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: <AppFuture at 0x14a6fb745b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211
parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: <AppFuture at 0x14a6fbd16450 state=pending>
parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: <AppFuture at 0x14a6fd0fca90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 4afd265e5a0681d6ee78e9654d528ec4
parsl.dataflow.memoization INFO: Task 213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b3713cf3-72ed-4885-8da1-2d66f53d608a_isr_947418_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b3713cf3-72ed-4885-8da1-2d66f53d608a_isr_947418_59.stderr
parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213
parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: <AppFuture at 0x14a6fd0fcb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214
parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: <AppFuture at 0x14a6ff04c290 state=pending>
parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: <AppFuture at 0x14a6ff04db50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 1092f459960700a43a9102dd6b5f094a
parsl.dataflow.memoization INFO: Task 216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d084e7dc-693f-41b3-a4ba-28a33ff67717_isr_947418_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d084e7dc-693f-41b3-a4ba-28a33ff67717_isr_947418_47.stderr
parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216
parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: <AppFuture at 0x14a6fbc9c5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217
parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: <AppFuture at 0x14a6fae36410 state=pending>
parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: <AppFuture at 0x14a6fae34f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 52aafc66efa5d142ca75b9c3b3c8dc7d
parsl.dataflow.memoization INFO: Task 219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/55185d5e-c44c-4350-9407-16a174b68a57_isr_947526_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/55185d5e-c44c-4350-9407-16a174b68a57_isr_947526_38.stderr
parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219
parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: <AppFuture at 0x14a6fbb2f990 state=pending>
parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220
parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: <AppFuture at 0x14a6fbc9dc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: <AppFuture at 0x14a6fb38ee50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 6599a44fba57a1b4c93a5e44c834e027
parsl.dataflow.memoization INFO: Task 222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ca918702-58c4-4c4b-95d8-7d81f6d1c26e_isr_947422_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ca918702-58c4-4c4b-95d8-7d81f6d1c26e_isr_947422_21.stderr
parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222
parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: <AppFuture at 0x14a6fb38f150 state=pending>
parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223
parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: <AppFuture at 0x14a6fb38d310 state=pending>
parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: <AppFuture at 0x14a6fbaa7910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 52434a50e159a1b9c29e5840a1737992
parsl.dataflow.memoization INFO: Task 225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4e87ca2b-79d6-4eac-b64b-a532dcf04f8d_isr_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4e87ca2b-79d6-4eac-b64b-a532dcf04f8d_isr_947523_2.stderr
parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225
parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: <AppFuture at 0x14a6fbaa4050 state=pending>
parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226
parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: <AppFuture at 0x14a6fbaa5c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: <AppFuture at 0x14a6fb83e790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 228 has memoization hash e18572624e84aefb471e0f0a75b549c7
parsl.dataflow.memoization INFO: Task 228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/95072ab7-8fbb-4a83-87c7-875b82e130dc_isr_947418_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/95072ab7-8fbb-4a83-87c7-875b82e130dc_isr_947418_1.stderr
parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228
parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: <AppFuture at 0x14a6fbaa7650 state=pending>
parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229
parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: <AppFuture at 0x14a6fb3d7050 state=pending>
parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: <AppFuture at 0x14a6fb3ad250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 2ce8e1f697ed2e2c0189c17bcfb2c587
parsl.dataflow.memoization INFO: Task 231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/23c85021-ddf3-4ae8-8f1c-e55603ea2dea_isr_947420_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/23c85021-ddf3-4ae8-8f1c-e55603ea2dea_isr_947420_36.stderr
parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231
parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: <AppFuture at 0x14a6fb3aea10 state=pending>
parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232
parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: <AppFuture at 0x14a6fb7cab90 state=pending>
parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: <AppFuture at 0x14a6fb606ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 8e215bf2a12a57664149d0eeaac5be53
parsl.dataflow.memoization INFO: Task 234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f1ed7b76-e25f-4b68-9dc5-d9a1a54229b8_isr_947526_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f1ed7b76-e25f-4b68-9dc5-d9a1a54229b8_isr_947526_4.stderr
parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234
parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: <AppFuture at 0x14a6fb6065d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235
parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: <AppFuture at 0x14a6fb605650 state=pending>
parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: <AppFuture at 0x14a6fb77ef50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 237 has memoization hash c9e7201be891a5f86e667410f459a45b
parsl.dataflow.memoization INFO: Task 237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c0ef73ae-6759-4a8b-b557-d0c0281970b2_isr_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c0ef73ae-6759-4a8b-b557-d0c0281970b2_isr_947420_2.stderr
parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237
parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: <AppFuture at 0x14a6fc1edb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238
parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: <AppFuture at 0x14a6fb6258d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: <AppFuture at 0x14a6fb60aa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 620533edd364fa989b469551ebc744fa
parsl.dataflow.memoization INFO: Task 240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f4f899b-c0ff-4c8a-b953-bf335f9ac109_isr_947420_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f4f899b-c0ff-4c8a-b953-bf335f9ac109_isr_947420_46.stderr
parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240
parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: <AppFuture at 0x14a6fb60ac10 state=pending>
parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241
parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: <AppFuture at 0x14a6fb4dd0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: <AppFuture at 0x14a6fb4df510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 243 has memoization hash f6cc3b90cc7d684c3b70a436d668bc17
parsl.dataflow.memoization INFO: Task 243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/1aaad9a1-4d74-4850-9bcf-a1f7dd44e2ed_isr_947422_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/1aaad9a1-4d74-4850-9bcf-a1f7dd44e2ed_isr_947422_36.stderr
parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243
parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: <AppFuture at 0x14a6fb4dcd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244
parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: <AppFuture at 0x14a6fabc4790 state=pending>
parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: <AppFuture at 0x14a6fabc5410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 5ebef4deae6185a87aff16465017eb04
parsl.dataflow.memoization INFO: Task 246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30320618-095e-46fa-a562-3fe85faf0da9_isr_947420_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30320618-095e-46fa-a562-3fe85faf0da9_isr_947420_23.stderr
parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246
parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: <AppFuture at 0x14a6fb4de710 state=pending>
parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247
parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: <AppFuture at 0x14a6facdc1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: <AppFuture at 0x14a6facdef50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 249 has memoization hash a4b1e6e0c2e3510a36b48dd742e28849
parsl.dataflow.memoization INFO: Task 249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/66de8d43-aef4-4c29-aa81-4d577ba8242e_isr_947528_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/66de8d43-aef4-4c29-aa81-4d577ba8242e_isr_947528_16.stderr
parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249
parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: <AppFuture at 0x14a6facdc810 state=pending>
parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250
parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: <AppFuture at 0x14a6faa36b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: <AppFuture at 0x14a6face4390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 92a7cece00aac9a5ad085c0dbb0491cd
parsl.dataflow.memoization INFO: Task 252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8a342e0e-0b63-412a-8f1e-cbd940529c28_isr_947418_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8a342e0e-0b63-412a-8f1e-cbd940529c28_isr_947418_16.stderr
parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252
parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: <AppFuture at 0x14a6face6950 state=pending>
parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253
parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: <AppFuture at 0x14a6ff0408d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: <AppFuture at 0x14a6ff042210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 255 has memoization hash aff8cee467ef3d4af4aac693d5434241
parsl.dataflow.memoization INFO: Task 255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/42d071ee-2b6d-4817-a039-9cb236c964da_isr_947528_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/42d071ee-2b6d-4817-a039-9cb236c964da_isr_947528_8.stderr
parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255
parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: <AppFuture at 0x14a6ff042190 state=pending>
parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256
parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: <AppFuture at 0x14a6fb38fd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: <AppFuture at 0x14a6ff00f890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 258 has memoization hash ee98e535ce7ba1975341a7e3f133d0b0
parsl.dataflow.memoization INFO: Task 258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b36ed276-870f-4838-8eaa-03b7600cc3c7_isr_947528_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b36ed276-870f-4838-8eaa-03b7600cc3c7_isr_947528_45.stderr
parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258
parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: <AppFuture at 0x14a6faac5810 state=pending>
parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259
parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: <AppFuture at 0x14a6ff00df50 state=pending>
parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: <AppFuture at 0x14a6ff00d750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 76eda758a0c1a2b146e68e3eec307d7a
parsl.dataflow.memoization INFO: Task 261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c84583a9-0200-4b1b-93af-424d56042ce4_isr_947526_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c84583a9-0200-4b1b-93af-424d56042ce4_isr_947526_57.stderr
parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261
parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: <AppFuture at 0x14a6ff00dd10 state=pending>
parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262
parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: <AppFuture at 0x14a6ff040390 state=pending>
parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: <AppFuture at 0x14a6faa02ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 264 has memoization hash dc6344ddc91dfd4211ffb9ebb6a6c78d
parsl.dataflow.memoization INFO: Task 264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9cc439b7-233e-4410-9342-92d50676c567_isr_947526_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9cc439b7-233e-4410-9342-92d50676c567_isr_947526_42.stderr
parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264
parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: <AppFuture at 0x14a6fab15f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265
parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: <AppFuture at 0x14a6faa02d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: <AppFuture at 0x14a6fa8175d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 766caba1caf75da452038f16823489ab
parsl.dataflow.memoization INFO: Task 267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/28f32647-caa3-47fc-8a6f-97ae950bc62c_isr_947528_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/28f32647-caa3-47fc-8a6f-97ae950bc62c_isr_947528_31.stderr
parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267
parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: <AppFuture at 0x14a6fa814ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268
parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: <AppFuture at 0x14a6fa827590 state=pending>
parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: <AppFuture at 0x14a6fc4f2110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 35db56a5f62d35e4c058bb242b1f77a4
parsl.dataflow.memoization INFO: Task 270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d3e162b3-7d8c-4cb2-8ab6-a244a59e4058_isr_947418_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d3e162b3-7d8c-4cb2-8ab6-a244a59e4058_isr_947418_29.stderr
parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270
parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: <AppFuture at 0x14a6fc4f27d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271
parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: <AppFuture at 0x14a6fc4f3310 state=pending>
parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: <AppFuture at 0x14a6ff008450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 3d639087b200327dc92a0fef77637a9f
parsl.dataflow.memoization INFO: Task 273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/3da3af2f-f3f4-49c6-94cd-4b86764a97f0_isr_947524_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/3da3af2f-f3f4-49c6-94cd-4b86764a97f0_isr_947524_53.stderr
parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273
parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: <AppFuture at 0x14a6ff0089d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274
parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: <AppFuture at 0x14a6ff00a110 state=pending>
parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: <AppFuture at 0x14a6ff00aad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 9d3b39d5403c2968085cb380e467f59b
parsl.dataflow.memoization INFO: Task 276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f009a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cad49c99-d089-4cf0-9fee-b85261707ee0_isr_947523_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cad49c99-d089-4cf0-9fee-b85261707ee0_isr_947523_47.stderr
parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276
parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: <AppFuture at 0x14a6ff00b8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277
parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: <AppFuture at 0x14a6fefc2450 state=pending>
parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: <AppFuture at 0x14a6ff02d2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 51efb2ef95aa6f6912c0e7f91760e87e
parsl.dataflow.memoization INFO: Task 279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bbd78b0-b53e-4bf8-9b3d-aeb6990066d7_isr_947420_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bbd78b0-b53e-4bf8-9b3d-aeb6990066d7_isr_947420_25.stderr
parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279
parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: <AppFuture at 0x14a6ff00b750 state=pending>
parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280
parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: <AppFuture at 0x14a6ff02ed10 state=pending>
parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: <AppFuture at 0x14a6ff004850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 282 has memoization hash fa148597e3bc233f1698475a7d5e0e0c
parsl.dataflow.memoization INFO: Task 282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e8bafed2-bfe4-45e6-aeb9-7be2643a1765_isr_947418_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e8bafed2-bfe4-45e6-aeb9-7be2643a1765_isr_947418_51.stderr
parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282
parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: <AppFuture at 0x14a6ff02db50 state=pending>
parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283
parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: <AppFuture at 0x14a6ff007cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: <AppFuture at 0x14a6ff007890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 05877509337712ba848e115404b5404d
parsl.dataflow.memoization INFO: Task 285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f9cf9043-618c-4099-8a4d-c6d870d9c1cc_isr_947418_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f9cf9043-618c-4099-8a4d-c6d870d9c1cc_isr_947418_10.stderr
parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285
parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: <AppFuture at 0x14a6ff0046d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286
parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: <AppFuture at 0x14a6fb2a9310 state=pending>
parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: <AppFuture at 0x14a6fb2ab110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 9d6c05285e1eea24c6d10ce13a7cb4ef
parsl.dataflow.memoization INFO: Task 288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8cb87310-6730-42bb-8977-e0bc8e9f70dc_isr_947420_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8cb87310-6730-42bb-8977-e0bc8e9f70dc_isr_947420_43.stderr
parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288
parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: <AppFuture at 0x14a6fb021990 state=pending>
parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289
parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: <AppFuture at 0x14a6fb021c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: <AppFuture at 0x14a6fd09bdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 46143bd2aca114be53d56d154c8cb9c8
parsl.dataflow.memoization INFO: Task 291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/96f56684-2f90-4643-9574-71f6f1eab4b3_isr_947524_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/96f56684-2f90-4643-9574-71f6f1eab4b3_isr_947524_8.stderr
parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291
parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: <AppFuture at 0x14a6fd09a850 state=pending>
parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292
parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: <AppFuture at 0x14a6fd09b490 state=pending>
parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: <AppFuture at 0x14a6faabccd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 73f3a6e3db26a83edff7115c85775bc4
parsl.dataflow.memoization INFO: Task 294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/66811516-e824-4465-aca2-9fb7cad5c3b9_isr_947523_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/66811516-e824-4465-aca2-9fb7cad5c3b9_isr_947523_19.stderr
parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294
parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: <AppFuture at 0x14a6faabfe50 state=pending>
parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295
parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: <AppFuture at 0x14a6fb523190 state=pending>
parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: <AppFuture at 0x14a6fb522590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 297 has memoization hash a7f612d38e1ef22ab92de24bf548bd43
parsl.dataflow.memoization INFO: Task 297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/41cf2400-f503-4fdb-89e5-403643656846_isr_947526_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/41cf2400-f503-4fdb-89e5-403643656846_isr_947526_55.stderr
parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297
parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: <AppFuture at 0x14a6fb5205d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298
parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: <AppFuture at 0x14a6fb521650 state=pending>
parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: <AppFuture at 0x14a6fc6dbf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 0e124c872966de2eea98e049c80bc84e
parsl.dataflow.memoization INFO: Task 300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f022a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/56732e0a-bf73-4ad4-8d6d-0b21a76a507b_isr_947526_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/56732e0a-bf73-4ad4-8d6d-0b21a76a507b_isr_947526_12.stderr
parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300
parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: <AppFuture at 0x14a6fc6d8b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301
parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: <AppFuture at 0x14a6fc6d9b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: <AppFuture at 0x14a6fc6ae850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 2f94f9e218d67fc22d57d383efdfa164
parsl.dataflow.memoization INFO: Task 303 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f025c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/691af0bd-3d3b-46c4-884b-fd8949fc9122_isr_947523_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/691af0bd-3d3b-46c4-884b-fd8949fc9122_isr_947523_31.stderr
parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303
parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: <AppFuture at 0x14a6fc6aff50 state=pending>
parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304
parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: <AppFuture at 0x14a6faa02fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: <AppFuture at 0x14a6fc742e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 290b17c6f1b73cc315c23149a4c85e2d
parsl.dataflow.memoization INFO: Task 306 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f028e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9e409e6a-1549-44c7-9c7f-397a83960109_isr_947523_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9e409e6a-1549-44c7-9c7f-397a83960109_isr_947523_22.stderr
parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306
parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: <AppFuture at 0x14a6fc743e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307
parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: <AppFuture at 0x14a6fc740710 state=pending>
parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: <AppFuture at 0x14a6fc4cb110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 7182c5a3b02e67adfc611b9415e1d296
parsl.dataflow.memoization INFO: Task 309 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/20fe30b2-8624-42b5-b48d-2e0c6841d1cc_isr_947418_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/20fe30b2-8624-42b5-b48d-2e0c6841d1cc_isr_947418_3.stderr
parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309
parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: <AppFuture at 0x14a6fc4cb150 state=pending>
parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310
parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: <AppFuture at 0x14a6fc4c9450 state=pending>
parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: <AppFuture at 0x14a701324e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 24e72061678ceb42326c748cde7416b8
parsl.dataflow.memoization INFO: Task 312 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/933ad3e0-53d8-4d37-8a7c-742a9d8f3da5_isr_947420_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/933ad3e0-53d8-4d37-8a7c-742a9d8f3da5_isr_947420_56.stderr
parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312
parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: <AppFuture at 0x14a701325d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313
parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: <AppFuture at 0x14a6fa89e950 state=pending>
parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: <AppFuture at 0x14a6fc4cabd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 315 has memoization hash db50166195a414d6c51f9b832546878c
parsl.dataflow.memoization INFO: Task 315 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e9e62966-cb3d-496a-9ecb-eb7737439bf9_isr_947526_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e9e62966-cb3d-496a-9ecb-eb7737439bf9_isr_947526_6.stderr
parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315
parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: <AppFuture at 0x14a6fc4009d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316
parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: <AppFuture at 0x14a6fc400b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: <AppFuture at 0x14a6fbbb6150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 318 has memoization hash e79ce336bf272608a5aac34a1c6f00fa
parsl.dataflow.memoization INFO: Task 318 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d27d7185-225c-4dc4-9315-bde423fcf938_isr_947420_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d27d7185-225c-4dc4-9315-bde423fcf938_isr_947420_62.stderr
parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318
parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: <AppFuture at 0x14a6fbbb6690 state=pending>
parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319
parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: <AppFuture at 0x14a6fae5d290 state=pending>
parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: <AppFuture at 0x14a70163f6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 321 has memoization hash be04bc1796e93f2fa898adc690210f4d
parsl.dataflow.memoization INFO: Task 321 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4a54b160-cdec-46c0-bb1a-3b7dba461183_isr_947420_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4a54b160-cdec-46c0-bb1a-3b7dba461183_isr_947420_51.stderr
parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321
parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: <AppFuture at 0x14a70163f650 state=pending>
parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322
parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: <AppFuture at 0x14a6fbfeb910 state=pending>
parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: <AppFuture at 0x14a6fbfe8710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 324 has memoization hash d95e82b09f7744811cc5d8b892610a7d
parsl.dataflow.memoization INFO: Task 324 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7b6167b8-e550-4de5-97b9-7fa01051b86e_isr_947523_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7b6167b8-e550-4de5-97b9-7fa01051b86e_isr_947523_30.stderr
parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324
parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: <AppFuture at 0x14a6fbfea950 state=pending>
parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325
parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: <AppFuture at 0x14a6fa931210 state=pending>
parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: <AppFuture at 0x14a6fa9334d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 327 has memoization hash e0aa2ee49a09133fee6cc1f3adddb3fd
parsl.dataflow.memoization INFO: Task 327 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/66a527a8-1cc4-4a5e-841c-a219517274af_isr_947418_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/66a527a8-1cc4-4a5e-841c-a219517274af_isr_947418_18.stderr
parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327
parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: <AppFuture at 0x14a6fb9737d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328
parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: <AppFuture at 0x14a6fbe28390 state=pending>
parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: <AppFuture at 0x14a6fbe2a650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 624ffe445aabce00c97c4b2b04be5542
parsl.dataflow.memoization INFO: Task 330 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5573ebb0-648e-4e74-a055-457b4079740f_isr_947422_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5573ebb0-648e-4e74-a055-457b4079740f_isr_947422_8.stderr
parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330
parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: <AppFuture at 0x14a6fbe29990 state=pending>
parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331
parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: <AppFuture at 0x14a6fbe29dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: <AppFuture at 0x14a6faa63bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 333 has memoization hash c92b6fa14b462286ae89677fde510da6
parsl.dataflow.memoization INFO: Task 333 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/feb4d085-3c17-4639-beac-dc3570e835c4_isr_947524_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/feb4d085-3c17-4639-beac-dc3570e835c4_isr_947524_34.stderr
parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333
parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: <AppFuture at 0x14a6fbe28750 state=pending>
parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334
parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: <AppFuture at 0x14a6fbb730d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: <AppFuture at 0x14a6fb378210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 336 has memoization hash c9d51f620f04eacad437278305ebd3dc
parsl.dataflow.memoization INFO: Task 336 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bce8f3f3-e731-47d9-88bb-143ef16fe8e1_isr_947418_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bce8f3f3-e731-47d9-88bb-143ef16fe8e1_isr_947418_58.stderr
parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336
parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: <AppFuture at 0x14a6fb37b2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337
parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: <AppFuture at 0x14a6fb94a750 state=pending>
parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: <AppFuture at 0x14a6fb949a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 339 has memoization hash ae56f8d158ecd2b470ed19f49e6c741e
parsl.dataflow.memoization INFO: Task 339 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/71e3df41-3bca-471c-9705-61297254aef0_isr_947524_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/71e3df41-3bca-471c-9705-61297254aef0_isr_947524_20.stderr
parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339
parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: <AppFuture at 0x14a6fb94ad90 state=pending>
parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340
parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: <AppFuture at 0x14a6fb886890 state=pending>
parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: <AppFuture at 0x14a6fb471d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 320c6fded4fe80a5fb49c4d1993e2f88
parsl.dataflow.memoization INFO: Task 342 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1c767594-9473-446d-b06b-287f51c8449a_isr_947418_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1c767594-9473-446d-b06b-287f51c8449a_isr_947418_25.stderr
parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342
parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: <AppFuture at 0x14a6fb4726d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343
parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: <AppFuture at 0x14a6fb424710 state=pending>
parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: <AppFuture at 0x14a6fb427950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 345 has memoization hash fe18fda7c10f8113fe3bec4009ff1c6c
parsl.dataflow.memoization INFO: Task 345 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/69d66532-c49f-408a-bfc3-3cc223b8a3a4_isr_947524_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/69d66532-c49f-408a-bfc3-3cc223b8a3a4_isr_947524_33.stderr
parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345
parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: <AppFuture at 0x14a6fb465b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346
parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: <AppFuture at 0x14a6fb2eded0 state=pending>
parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: <AppFuture at 0x14a6fb2efcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 7c7177f6c21dcad9e4171a6b7e41eae5
parsl.dataflow.memoization INFO: Task 348 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d15e6e19-dca4-42c1-a914-86452e3591e7_isr_947418_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d15e6e19-dca4-42c1-a914-86452e3591e7_isr_947418_15.stderr
parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348
parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: <AppFuture at 0x14a6fb2ed2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349
parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: <AppFuture at 0x14a6fac70ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: <AppFuture at 0x14a6faf96c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 351 has memoization hash aa10a5b269e51d3d79eae7ae7603cde8
parsl.dataflow.memoization INFO: Task 351 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/372bc865-ea4b-40de-b127-3f7c06a3e12b_isr_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/372bc865-ea4b-40de-b127-3f7c06a3e12b_isr_947422_2.stderr
parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351
parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: <AppFuture at 0x14a6fa8df810 state=pending>
parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352
parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: <AppFuture at 0x14a6fb258850 state=pending>
parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: <AppFuture at 0x14a6fb25b190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 9f6a6e679273cddece14ee69527b0d3d
parsl.dataflow.memoization INFO: Task 354 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/589bc693-3130-4169-8e85-24ce137a9cd5_isr_947526_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/589bc693-3130-4169-8e85-24ce137a9cd5_isr_947526_54.stderr
parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354
parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: <AppFuture at 0x14a6fb2585d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355
parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: <AppFuture at 0x14a6fb25a850 state=pending>
parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: <AppFuture at 0x14a70131fc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 4525d1f36eaa6bc23e1eec455304db64
parsl.dataflow.memoization INFO: Task 357 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7667c897-246d-4f9f-9303-0c8388ac2b29_isr_947523_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7667c897-246d-4f9f-9303-0c8388ac2b29_isr_947523_42.stderr
parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357
parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: <AppFuture at 0x14a70131cb10 state=pending>
parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358
parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: <AppFuture at 0x14a6fb189890 state=pending>
parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: <AppFuture at 0x14a6fb1a1a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 865a97b2466c858f4dc95bbf9129b919
parsl.dataflow.memoization INFO: Task 360 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f0328ea-977b-49d1-85ed-a941486f59c8_isr_947420_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f0328ea-977b-49d1-85ed-a941486f59c8_isr_947420_11.stderr
parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360
parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: <AppFuture at 0x14a70131de50 state=pending>
parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361
parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: <AppFuture at 0x14a6fb08c150 state=pending>
parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: <AppFuture at 0x14a6fadde090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 363 has memoization hash ed013c4a6d237e68a93f60ea9e20aec8
parsl.dataflow.memoization INFO: Task 363 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/38b1a1eb-7105-40d1-91f3-f5fd08a041db_isr_947420_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/38b1a1eb-7105-40d1-91f3-f5fd08a041db_isr_947420_30.stderr
parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363
parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: <AppFuture at 0x14a6fb08ea50 state=pending>
parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364
parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: <AppFuture at 0x14a6fb37b9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: <AppFuture at 0x14a6fad81b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 366 has memoization hash a6d12ffa4ffa7b79d4b56e72b4890df5
parsl.dataflow.memoization INFO: Task 366 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7fd26538-ba62-4222-ab52-8466592e578f_isr_947528_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7fd26538-ba62-4222-ab52-8466592e578f_isr_947528_54.stderr
parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366
parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: <AppFuture at 0x14a6fad83d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367
parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: <AppFuture at 0x14a6fabd4950 state=pending>
parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: <AppFuture at 0x14a6fabd6f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 369 has memoization hash fde419ef2456905b63e46753d94f8e9b
parsl.dataflow.memoization INFO: Task 369 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30a8834b-8fba-44fb-b982-9e4565c780e7_isr_947420_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30a8834b-8fba-44fb-b982-9e4565c780e7_isr_947420_53.stderr
parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369
parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: <AppFuture at 0x14a6fac91a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370
parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: <AppFuture at 0x14a6faafd1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: <AppFuture at 0x14a70162d690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 303cc821a40fdec0cb75316cfeb28af1
parsl.dataflow.memoization INFO: Task 372 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6e403693-3e6a-4f0d-becd-5df99a83504a_isr_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6e403693-3e6a-4f0d-becd-5df99a83504a_isr_947528_2.stderr
parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372
parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: <AppFuture at 0x14a70162f290 state=pending>
parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373
parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: <AppFuture at 0x14a70162ef90 state=pending>
parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: <AppFuture at 0x14a6fa9e7c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 375 has memoization hash a08a639dea99ce796931cdb2080755b4
parsl.dataflow.memoization INFO: Task 375 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/cfe58559-c8ce-4124-98fe-b8e0ed65df7b_isr_947524_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/cfe58559-c8ce-4124-98fe-b8e0ed65df7b_isr_947524_58.stderr
parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375
parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: <AppFuture at 0x14a6fa6a8810 state=pending>
parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376
parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: <AppFuture at 0x14a6fa9e6210 state=pending>
parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: <AppFuture at 0x14a6ff277150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 259f462893160238b91d8e94e3cfb21f
parsl.dataflow.memoization INFO: Task 378 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb533f57-2947-4a28-ab0c-13288f7ed778_isr_947418_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb533f57-2947-4a28-ab0c-13288f7ed778_isr_947418_13.stderr
parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378
parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: <AppFuture at 0x14a6ff277450 state=pending>
parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379
parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: <AppFuture at 0x14a6ff277610 state=pending>
parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: <AppFuture at 0x14a6fa6b8210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 099bf0cedb54806d354de9ead9dfb29a
parsl.dataflow.memoization INFO: Task 381 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47c4ffca-3c6b-49a9-aa71-938738fcdb8f_isr_947420_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47c4ffca-3c6b-49a9-aa71-938738fcdb8f_isr_947420_24.stderr
parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381
parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: <AppFuture at 0x14a6fa6c1590 state=pending>
parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382
parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: <AppFuture at 0x14a6fa786990 state=pending>
parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: <AppFuture at 0x14a6fa768110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 2f2b8c352111f37add915af2dc04a785
parsl.dataflow.memoization INFO: Task 384 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/61acb7f8-1fab-4385-b1c5-dfffaab3ecc2_isr_947422_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/61acb7f8-1fab-4385-b1c5-dfffaab3ecc2_isr_947422_54.stderr
parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384
parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: <AppFuture at 0x14a6fa771b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385
parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: <AppFuture at 0x14a6fceff290 state=pending>
parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: <AppFuture at 0x14a6fcefe650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 5ffbfe82670e5b77deb492409e413f9a
parsl.dataflow.memoization INFO: Task 387 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3492cd23-9da0-4650-8fd4-17579da37677_isr_947420_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3492cd23-9da0-4650-8fd4-17579da37677_isr_947420_42.stderr
parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387
parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: <AppFuture at 0x14a6fcefeb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388
parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: <AppFuture at 0x14a6fc4ad7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: <AppFuture at 0x14a6fc4afe90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 390 has memoization hash d0a563fb0fff2374d899dad8dc9cab58
parsl.dataflow.memoization INFO: Task 390 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f500e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c60fd252-54c9-41ef-921b-5bb9e4ee9984_isr_947420_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c60fd252-54c9-41ef-921b-5bb9e4ee9984_isr_947420_22.stderr
parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390
parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: <AppFuture at 0x14a6fc4adf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391
parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: <AppFuture at 0x14a6fc2526d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: <AppFuture at 0x14a6fc2529d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 44992cf8cbf14256043223b2b678a920
parsl.dataflow.memoization INFO: Task 393 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c585f076-afec-4b32-b239-d81c89a4cf8b_isr_947418_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c585f076-afec-4b32-b239-d81c89a4cf8b_isr_947418_42.stderr
parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393
parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: <AppFuture at 0x14a6fc252350 state=pending>
parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394
parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: <AppFuture at 0x14a6fc253c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: <AppFuture at 0x14a70bb3fa10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 396 has memoization hash bd169110583857134280e70d1ba0eaae
parsl.dataflow.memoization INFO: Task 396 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7e818045-3f79-40e7-aa11-5e1444cc01f8_isr_947420_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7e818045-3f79-40e7-aa11-5e1444cc01f8_isr_947420_34.stderr
parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396
parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: <AppFuture at 0x14a70bb3f850 state=pending>
parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397
parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: <AppFuture at 0x14a70bb3e450 state=pending>
parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: <AppFuture at 0x14a70bb3df50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 399 has memoization hash e084bf7d12d7db9807c9d2bc6f216c8f
parsl.dataflow.memoization INFO: Task 399 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/396b2ba5-a407-4335-862c-cfd4f6014fa0_isr_947523_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/396b2ba5-a407-4335-862c-cfd4f6014fa0_isr_947523_8.stderr
parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399
parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: <AppFuture at 0x14a70bb3ee90 state=pending>
parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400
parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: <AppFuture at 0x14a70bb3f210 state=pending>
parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: <AppFuture at 0x14a70bb3c610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 6aac33d612fd9b4b38cfd7f78091cdac
parsl.dataflow.memoization INFO: Task 402 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0b14d0f5-cd5a-4954-9b21-5196a497bec9_isr_947523_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0b14d0f5-cd5a-4954-9b21-5196a497bec9_isr_947523_11.stderr
parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402
parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: <AppFuture at 0x14a70bb3c110 state=pending>
parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403
parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: <AppFuture at 0x14a6fa7cffd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: <AppFuture at 0x14a70be3ad90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 405 has memoization hash e74f9122e661ed5c71fea2b3e3315ecd
parsl.dataflow.memoization INFO: Task 405 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/468b2559-d83b-4e10-8114-4fd6dd33d8db_isr_947422_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/468b2559-d83b-4e10-8114-4fd6dd33d8db_isr_947422_26.stderr
parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405
parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: <AppFuture at 0x14a70be97050 state=pending>
parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406
parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: <AppFuture at 0x14a70be396d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: <AppFuture at 0x14a70be3bf50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 20bbe3894d9fd3af63e1251b48e33bc3
parsl.dataflow.memoization INFO: Task 408 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5ffd17d9-42d7-4446-88fd-e65f0cbec730_isr_947526_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5ffd17d9-42d7-4446-88fd-e65f0cbec730_isr_947526_28.stderr
parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408
parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: <AppFuture at 0x14a70be397d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409
parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: <AppFuture at 0x14a6fedcee10 state=pending>
parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: <AppFuture at 0x14a6fef40090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 27686ce769305482e7080c6e57800857
parsl.dataflow.memoization INFO: Task 411 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f516c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b1f2df3a-1d41-4d56-82f6-d985ca66bea7_isr_947523_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b1f2df3a-1d41-4d56-82f6-d985ca66bea7_isr_947523_20.stderr
parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411
parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: <AppFuture at 0x14a6fedcc410 state=pending>
parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412
parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: <AppFuture at 0x14a6fef41390 state=pending>
parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: <AppFuture at 0x14a6fef40c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 5d284665a73e787da412fc8954ec8f09
parsl.dataflow.memoization INFO: Task 414 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/73d7a956-273e-4057-a7b3-c186dc2286e5_isr_947526_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/73d7a956-273e-4057-a7b3-c186dc2286e5_isr_947526_44.stderr
parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414
parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: <AppFuture at 0x14a6fef6f510 state=pending>
parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415
parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: <AppFuture at 0x14a6fef6cb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: <AppFuture at 0x14a6fef6c390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 417 has memoization hash feadaa1d6af92f8282decfcc2e19f469
parsl.dataflow.memoization INFO: Task 417 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b6e21d70-5f0c-431a-9493-37bc701ad0ca_isr_947528_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b6e21d70-5f0c-431a-9493-37bc701ad0ca_isr_947528_32.stderr
parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417
parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: <AppFuture at 0x14a6fef6f7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418
parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: <AppFuture at 0x14a6fef6ebd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: <AppFuture at 0x14a701794590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 14750067423c24d5525b7f4da21130f1
parsl.dataflow.memoization INFO: Task 420 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a44530df-6d8a-4803-9445-379becfd8990_isr_947418_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a44530df-6d8a-4803-9445-379becfd8990_isr_947418_32.stderr
parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420
parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: <AppFuture at 0x14a6fb94ac50 state=pending>
parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421
parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: <AppFuture at 0x14a7005521d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: <AppFuture at 0x14a701618910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 423 has memoization hash b1dce8b1920247ee8540f2bb2d399947
parsl.dataflow.memoization INFO: Task 423 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/472b2159-067a-4055-a02a-f402c27d6729_isr_947422_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/472b2159-067a-4055-a02a-f402c27d6729_isr_947422_45.stderr
parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423
parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: <AppFuture at 0x14a70161ad10 state=pending>
parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424
parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: <AppFuture at 0x14a70161a590 state=pending>
parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: <AppFuture at 0x14a70bb03fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 426 has memoization hash ad4f12b82457444c988ef5d4bc249743
parsl.dataflow.memoization INFO: Task 426 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/943e6b8b-5d28-4d27-92b6-bf0f6d63a08f_isr_947420_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/943e6b8b-5d28-4d27-92b6-bf0f6d63a08f_isr_947420_3.stderr
parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426
parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: <AppFuture at 0x14a701619710 state=pending>
parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427
parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: <AppFuture at 0x14a70bb03090 state=pending>
parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: <AppFuture at 0x14a70bb02610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 73c5ff22c9cb2f9abbb3a851a7ab49d8
parsl.dataflow.memoization INFO: Task 429 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/9f03f79b-c880-4a5f-8aaa-a3ae442016db_isr_947422_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/9f03f79b-c880-4a5f-8aaa-a3ae442016db_isr_947422_37.stderr
parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429
parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: <AppFuture at 0x14a70bb02910 state=pending>
parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430
parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: <AppFuture at 0x14a70bb01bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: <AppFuture at 0x14a70bb00c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 432 has memoization hash e83ca24da964101c455697be7298ec17
parsl.dataflow.memoization INFO: Task 432 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8b43018e-e111-406e-8936-4421ae7ac670_isr_947420_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8b43018e-e111-406e-8936-4421ae7ac670_isr_947420_45.stderr
parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432
parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: <AppFuture at 0x14a70bb00790 state=pending>
parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433
parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: <AppFuture at 0x14a70bba7c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: <AppFuture at 0x14a70bba73d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 435 has memoization hash afb4636fcb6049e9c8b2c4fdb6a61151
parsl.dataflow.memoization INFO: Task 435 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/a722a0e6-3790-4aca-ab2a-c281e41d538c_isr_947524_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/a722a0e6-3790-4aca-ab2a-c281e41d538c_isr_947524_15.stderr
parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435
parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: <AppFuture at 0x14a70bba7210 state=pending>
parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436
parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: <AppFuture at 0x14a70bba6890 state=pending>
parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: <AppFuture at 0x14a70bba5ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 9ff3113b98e3ee0b40591c6c2a01b769
parsl.dataflow.memoization INFO: Task 438 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f532e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e260a333-e212-48d2-91d7-97c335ed633f_isr_947526_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e260a333-e212-48d2-91d7-97c335ed633f_isr_947526_3.stderr
parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438
parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: <AppFuture at 0x14a70bba6a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439
parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: <AppFuture at 0x14a70bba5010 state=pending>
parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: <AppFuture at 0x14a70bba41d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 88970bf204c268c34c8dfcc068b01ce5
parsl.dataflow.memoization INFO: Task 441 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f7428d9c-2995-43ca-8c2c-dd2da17ccb7b_isr_947528_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f7428d9c-2995-43ca-8c2c-dd2da17ccb7b_isr_947528_21.stderr
parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441
parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: <AppFuture at 0x14a70bba4550 state=pending>
parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442
parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: <AppFuture at 0x14a70b3cf810 state=pending>
parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: <AppFuture at 0x14a70b3cf350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 612aaa34eb289262b1d84e240d029d29
parsl.dataflow.memoization INFO: Task 444 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7db810a3-a9bb-46c6-92a6-36a5198913c4_isr_947422_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7db810a3-a9bb-46c6-92a6-36a5198913c4_isr_947422_46.stderr
parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444
parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: <AppFuture at 0x14a70b3cf150 state=pending>
parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445
parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: <AppFuture at 0x14a70b3ce050 state=pending>
parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: <AppFuture at 0x14a70b3cd8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 6cde68f7becaaa1cb60b63ec435eec9a
parsl.dataflow.memoization INFO: Task 447 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b6da41f9-9d30-44b0-95cf-c14030a80468_isr_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b6da41f9-9d30-44b0-95cf-c14030a80468_isr_947422_61.stderr
parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447
parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: <AppFuture at 0x14a70b3cd850 state=pending>
parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448
parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: <AppFuture at 0x14a70b3cd050 state=pending>
parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: <AppFuture at 0x14a70b3cc190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 32a74d13485f94bf315296079b932c6c
parsl.dataflow.memoization INFO: Task 450 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e8ad685-80e6-4e93-a13f-cacd1efef590_isr_947420_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e8ad685-80e6-4e93-a13f-cacd1efef590_isr_947420_48.stderr
parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450
parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: <AppFuture at 0x14a6fc743850 state=pending>
parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451
parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: <AppFuture at 0x14a70bbfb0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: <AppFuture at 0x14a70bbfabd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 453 has memoization hash c00b551246051213f5912cacc25fa6bc
parsl.dataflow.memoization INFO: Task 453 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c5a899d-a38f-40ba-ac43-a10c2a684460_isr_947523_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c5a899d-a38f-40ba-ac43-a10c2a684460_isr_947523_27.stderr
parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453
parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: <AppFuture at 0x14a70bbfbad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454
parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: <AppFuture at 0x14a70bbf9910 state=pending>
parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: <AppFuture at 0x14a70bbf9150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 456 has memoization hash b7e45bcee568158002ccea095cb568c5
parsl.dataflow.memoization INFO: Task 456 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/4f3173f8-1b27-43bf-9839-f61007b18c51_isr_947422_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/4f3173f8-1b27-43bf-9839-f61007b18c51_isr_947422_10.stderr
parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456
parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: <AppFuture at 0x14a70bbfa0d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457
parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: <AppFuture at 0x14a70bbf8150 state=pending>
parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: <AppFuture at 0x14a70bbfadd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 459 has memoization hash 833d202ecc71fca1447a9097a5181c3e
parsl.dataflow.memoization INFO: Task 459 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/31a82a45-849a-4e38-9b2b-1238e594f89d_isr_947526_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/31a82a45-849a-4e38-9b2b-1238e594f89d_isr_947526_25.stderr
parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459
parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: <AppFuture at 0x14a6fa95db50 state=pending>
parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460
parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: <AppFuture at 0x14a6fef8b250 state=pending>
parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: <AppFuture at 0x14a6fef8bad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 462 has memoization hash d2c220f4923c2bce3a36abe1e95422bc
parsl.dataflow.memoization INFO: Task 462 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cd8a868-2e99-4de7-9064-815c240abb3a_isr_947418_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cd8a868-2e99-4de7-9064-815c240abb3a_isr_947418_43.stderr
parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462
parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: <AppFuture at 0x14a6fef8b750 state=pending>
parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463
parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: <AppFuture at 0x14a6fef8aa10 state=pending>
parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: <AppFuture at 0x14a6fef89f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 8744365a5dfb8a23dc46a7f2450874d2
parsl.dataflow.memoization INFO: Task 465 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/a24136e6-b29a-46aa-900d-158bdec834ff_isr_947523_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/a24136e6-b29a-46aa-900d-158bdec834ff_isr_947523_45.stderr
parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465
parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: <AppFuture at 0x14a6fef8a250 state=pending>
parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466
parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: <AppFuture at 0x14a6fef892d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: <AppFuture at 0x14a6fef88150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 5cc5f236e6332122fa968136dae4c40c
parsl.dataflow.memoization INFO: Task 468 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e1957a0-c51d-4ba6-a9bb-d849c2dc2d4c_isr_947420_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e1957a0-c51d-4ba6-a9bb-d849c2dc2d4c_isr_947420_26.stderr
parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468
parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: <AppFuture at 0x14a6fef890d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469
parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: <AppFuture at 0x14a6ff06d250 state=pending>
parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: <AppFuture at 0x14a6ff06de90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 9238ff43e76bab4a22b147aa28882094
parsl.dataflow.memoization INFO: Task 471 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4f262436-003b-4e1a-af9a-5a95fc51799d_isr_947524_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4f262436-003b-4e1a-af9a-5a95fc51799d_isr_947524_27.stderr
parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471
parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: <AppFuture at 0x14a6ff06e2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472
parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: <AppFuture at 0x14a6ff06e1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: <AppFuture at 0x14a6fef5ad50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 474 has memoization hash d746b1a5ae4039545533243a11fc28b2
parsl.dataflow.memoization INFO: Task 474 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1921d2ea-b283-4ebe-a684-c5f5a10a74c1_isr_947526_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1921d2ea-b283-4ebe-a684-c5f5a10a74c1_isr_947526_11.stderr
parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474
parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: <AppFuture at 0x14a6ff06eed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475
parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: <AppFuture at 0x14a6fef5bc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: <AppFuture at 0x14a6fef4cc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 477 has memoization hash edf6917f329c85af1451b45049132054
parsl.dataflow.memoization INFO: Task 477 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8d0e1e5b-9133-44f9-b7d7-98212a7422df_isr_947528_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8d0e1e5b-9133-44f9-b7d7-98212a7422df_isr_947528_59.stderr
parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477
parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: <AppFuture at 0x14a6ff06e310 state=pending>
parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478
parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: <AppFuture at 0x14a6fef4f7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: <AppFuture at 0x14a6fef4f050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 5c5f52a4ebca10b548f112cbb2158ec0
parsl.dataflow.memoization INFO: Task 480 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb3a7c28-211c-4016-af61-911ea573e9fa_isr_947418_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb3a7c28-211c-4016-af61-911ea573e9fa_isr_947418_60.stderr
parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480
parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: <AppFuture at 0x14a6fef4f4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481
parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: <AppFuture at 0x14a6fef4d9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: <AppFuture at 0x14a6fef4d410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 483 has memoization hash f891459501c74a70220f19f8c58455fe
parsl.dataflow.memoization INFO: Task 483 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/5c758297-6fd7-475c-a5a8-f194e9a68ff3_isr_947528_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/5c758297-6fd7-475c-a5a8-f194e9a68ff3_isr_947528_53.stderr
parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483
parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: <AppFuture at 0x14a6fef4cf50 state=pending>
parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484
parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: <AppFuture at 0x14a6fa50afd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: <AppFuture at 0x14a6fa50bd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 486 has memoization hash f0a3e651b5f7ce3acc7527283a5ce25f
parsl.dataflow.memoization INFO: Task 486 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/e50fa31f-bede-4b47-b200-5d202a6c3085_isr_947422_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/e50fa31f-bede-4b47-b200-5d202a6c3085_isr_947422_35.stderr
parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486
parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: <AppFuture at 0x14a6fa509e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487
parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: <AppFuture at 0x14a6fa6e4b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: <AppFuture at 0x14a6fa69a410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 6672dad07a435243c87bdfce911dd9fc
parsl.dataflow.memoization INFO: Task 489 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1f06fbd2-1c75-462a-8826-693feb81b9b1_isr_947523_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1f06fbd2-1c75-462a-8826-693feb81b9b1_isr_947523_9.stderr
parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489
parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: <AppFuture at 0x14a6fa6e5ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490
parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: <AppFuture at 0x14a6fa69aa10 state=pending>
parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: <AppFuture at 0x14a6fa73fe10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 0112de60fc483748bfdc06373cb5295c
parsl.dataflow.memoization INFO: Task 492 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/53cbe677-7f32-455c-a258-7b35ac69e014_isr_947418_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/53cbe677-7f32-455c-a258-7b35ac69e014_isr_947418_34.stderr
parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492
parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: <AppFuture at 0x14a6fa73da90 state=pending>
parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493
parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: <AppFuture at 0x14a6fa4bac10 state=pending>
parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: <AppFuture at 0x14a6fa779710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 6ba62f37dcd9112eac469ac6b00eef6e
parsl.dataflow.memoization INFO: Task 495 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/998f989c-adcd-4a07-bba4-12acda02faa9_isr_947523_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/998f989c-adcd-4a07-bba4-12acda02faa9_isr_947523_52.stderr
parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495
parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: <AppFuture at 0x14a6fa77a910 state=pending>
parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496
parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: <AppFuture at 0x14a6fa5d7cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: <AppFuture at 0x14a6fa67d790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 498 has memoization hash e6e230a925055b5b74afa18f04947f61
parsl.dataflow.memoization INFO: Task 498 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/c9963555-260d-47b6-930e-c7b49302ed2a_isr_947523_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/c9963555-260d-47b6-930e-c7b49302ed2a_isr_947523_50.stderr
parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498
parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: <AppFuture at 0x14a6face4490 state=pending>
parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499
parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: <AppFuture at 0x14a6fa66efd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: <AppFuture at 0x14a70138a550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 02f99a6dc215bd7f9bd29929c716f897
parsl.dataflow.memoization INFO: Task 501 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/44839bb4-e012-475a-897e-874592835039_isr_947528_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/44839bb4-e012-475a-897e-874592835039_isr_947528_1.stderr
parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501
parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: <AppFuture at 0x14a6fa770b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502
parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: <AppFuture at 0x14a701389510 state=pending>
parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: <AppFuture at 0x14a7013890d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 504 has memoization hash cd64cab7dc4196a448e8eb7032049128
parsl.dataflow.memoization INFO: Task 504 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75584bd1-9c2e-41ac-a541-7355daecbdd4_isr_947524_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75584bd1-9c2e-41ac-a541-7355daecbdd4_isr_947524_13.stderr
parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504
parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: <AppFuture at 0x14a701389690 state=pending>
parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505
parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: <AppFuture at 0x14a70bb32cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: <AppFuture at 0x14a70bb32710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 46819fe6c1157c3c90f4f81fe71a0db8
parsl.dataflow.memoization INFO: Task 507 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e56fb22a-f7f2-4c59-8d83-31d682f2e852_isr_947526_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e56fb22a-f7f2-4c59-8d83-31d682f2e852_isr_947526_37.stderr
parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507
parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: <AppFuture at 0x14a70bb32190 state=pending>
parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508
parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: <AppFuture at 0x14a70bb31210 state=pending>
parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: <AppFuture at 0x14a70bb30d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 93f5e3e2b55e51b27fd0f04bcac5642b
parsl.dataflow.memoization INFO: Task 510 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c69df377-7c21-4737-8abe-ff0d7093716a_isr_947528_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c69df377-7c21-4737-8abe-ff0d7093716a_isr_947528_7.stderr
parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510
parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: <AppFuture at 0x14a70bb31a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511
parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: <AppFuture at 0x14a70bb30250 state=pending>
parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: <AppFuture at 0x14a6fb045b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 513 has memoization hash c6ab2cb129902d1957b622d5ef9ffa88
parsl.dataflow.memoization INFO: Task 513 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/61923aa0-ab2e-4512-ac53-a5dbf51982ba_isr_947526_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/61923aa0-ab2e-4512-ac53-a5dbf51982ba_isr_947526_49.stderr
parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513
parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: <AppFuture at 0x14a6fb045410 state=pending>
parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514
parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: <AppFuture at 0x14a70bb32010 state=pending>
parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: <AppFuture at 0x14a6fafb9b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 0ec0b31351fd84e510b149ff486c7376
parsl.dataflow.memoization INFO: Task 516 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c7c7a9b-8116-46d4-b4f4-7163592db4ce_isr_947523_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c7c7a9b-8116-46d4-b4f4-7163592db4ce_isr_947523_24.stderr
parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516
parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: <AppFuture at 0x14a6fafbac10 state=pending>
parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517
parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: <AppFuture at 0x14a6fafbad90 state=pending>
parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: <AppFuture at 0x14a6fa7a1e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 3fc32f49484edef966edc00dd004b3b3
parsl.dataflow.memoization INFO: Task 519 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4063502f-a7eb-45c3-b8f8-04a58e0f5d85_isr_947523_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4063502f-a7eb-45c3-b8f8-04a58e0f5d85_isr_947523_26.stderr
parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519
parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: <AppFuture at 0x14a6fafb9c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520
parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: <AppFuture at 0x14a6fa677510 state=pending>
parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: <AppFuture at 0x14a6fa677410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 4662a96a3f583d3bc02bf29e9a8726fd
parsl.dataflow.memoization INFO: Task 522 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b4a52109-2dbb-4f10-b9ea-e3c08f4a930c_isr_947422_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b4a52109-2dbb-4f10-b9ea-e3c08f4a930c_isr_947422_12.stderr
parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522
parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: <AppFuture at 0x14a6fa6741d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523
parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: <AppFuture at 0x14a6fa4b1c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: <AppFuture at 0x14a6fa4e2a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 388b6971821e65d36f18b832c464b3d7
parsl.dataflow.memoization INFO: Task 525 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/49e0ba86-4807-41c3-938f-ddf4ee541d94_isr_947418_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/49e0ba86-4807-41c3-938f-ddf4ee541d94_isr_947418_17.stderr
parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525
parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: <AppFuture at 0x14a6fa4b0a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526
parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: <AppFuture at 0x14a6fa4e0590 state=pending>
parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: <AppFuture at 0x14a6fa4e15d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 9caa394d86abfd5d05468810f7fcbea4
parsl.dataflow.memoization INFO: Task 528 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d9e0820a-5adc-4640-ab5b-7b6b2ec519b3_isr_947526_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d9e0820a-5adc-4640-ab5b-7b6b2ec519b3_isr_947526_19.stderr
parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528
parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: <AppFuture at 0x14a6fa4e3d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529
parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: <AppFuture at 0x14a6fa51b910 state=pending>
parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: <AppFuture at 0x14a6fa51a950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 3fd432c48692e36827152c32b3174df1
parsl.dataflow.memoization INFO: Task 531 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b81c1d71-db3c-4cee-a84b-74eb18a80452_isr_947528_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b81c1d71-db3c-4cee-a84b-74eb18a80452_isr_947528_52.stderr
parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531
parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: <AppFuture at 0x14a6fa519a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532
parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: <AppFuture at 0x14a6fa51ba10 state=pending>
parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: <AppFuture at 0x14a6fa52f050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 2f278eb2b5ea2bd726ad015e1f68f2c6
parsl.dataflow.memoization INFO: Task 534 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
2024-03-14 07:25:38 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c06eecdd-32ee-4de6-9127-c1c7222a5ca0_isr_947526_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c06eecdd-32ee-4de6-9127-c1c7222a5ca0_isr_947526_32.stderr
parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534
parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: <AppFuture at 0x14a6fe780dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535
parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: <AppFuture at 0x14a6fa511350 state=pending>
parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: <AppFuture at 0x14a6fa512f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 97cd14b5a3273be07cd9586a1ae0d48f
parsl.dataflow.memoization INFO: Task 537 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c61815eb-df2a-4eac-b9c2-adadebe4966d_isr_947526_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c61815eb-df2a-4eac-b9c2-adadebe4966d_isr_947526_50.stderr
parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537
parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: <AppFuture at 0x14a6fa5119d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538
parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: <AppFuture at 0x14a6fa512610 state=pending>
parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: <AppFuture at 0x14a6fa540110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 540 has memoization hash da9d5b5b1482d42bd1bcf0ca223162f6
parsl.dataflow.memoization INFO: Task 540 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/4f606d98-a3cf-4eb1-b489-ab62d468b66f_isr_947418_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/4f606d98-a3cf-4eb1-b489-ab62d468b66f_isr_947418_9.stderr
parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540
parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: <AppFuture at 0x14a6fb72a090 state=pending>
parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541
parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: <AppFuture at 0x14a6fa543810 state=pending>
parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: <AppFuture at 0x14a6fa541f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 3b47599671b02a8ef99614756df59d03
parsl.dataflow.memoization INFO: Task 543 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c3428dc4-f56c-48eb-97f3-3b3be4208484_isr_947418_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c3428dc4-f56c-48eb-97f3-3b3be4208484_isr_947418_7.stderr
parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543
parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: <AppFuture at 0x14a6fa52c590 state=pending>
parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544
parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: <AppFuture at 0x14a6fa4f4090 state=pending>
parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: <AppFuture at 0x14a6fa4f6bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 546 has memoization hash ec4eddd1d73e4d6c931f76fdaa78f5fe
parsl.dataflow.memoization INFO: Task 546 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6db09c80-333c-4981-8c41-b03300f0bffb_isr_947526_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6db09c80-333c-4981-8c41-b03300f0bffb_isr_947526_47.stderr
parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546
parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: <AppFuture at 0x14a70bb03710 state=pending>
parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547
parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: <AppFuture at 0x14a6fa54c950 state=pending>
parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: <AppFuture at 0x14a6fa54ec50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 549 has memoization hash d1e126313d3c1aa379d2414c6739bdd5
parsl.dataflow.memoization INFO: Task 549 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fb7cf08c-cdb8-461a-ac29-97afa6e7fb16_isr_947418_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fb7cf08c-cdb8-461a-ac29-97afa6e7fb16_isr_947418_56.stderr
parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549
parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: <AppFuture at 0x14a6fa54e6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550
parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: <AppFuture at 0x14a6fc6d9190 state=pending>
parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: <AppFuture at 0x14a6fd06ec10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 552 has memoization hash e278472c4d0be944f03b96b96e0b5acf
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 184
parsl.dataflow.memoization INFO: Task 552 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 1/0 running/pending blocks, and 0 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 45 more blocks
parsl.executors.status_handling INFO: Scaling out by 45 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-14 07:25:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-14 07:25:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 07:25:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/e53968f1-e465-4248-9042-c85e0109fa00_isr_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/e53968f1-e465-4248-9042-c85e0109fa00_isr_947524_2.stderr
parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552
parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: <AppFuture at 0x14a6fa9e6010 state=pending>
parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553
parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: <AppFuture at 0x14a6fa572390 state=pending>
parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: <AppFuture at 0x14a6fa5725d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 555 has memoization hash eede28835262f068cf1f0727326f9c65
parsl.dataflow.memoization INFO: Task 555 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0a400bf3-7abc-4f6f-ac42-158d3eb89113_isr_947418_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0a400bf3-7abc-4f6f-ac42-158d3eb89113_isr_947418_5.stderr
parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555
parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: <AppFuture at 0x14a6fa571dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556
parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: <AppFuture at 0x14a6f8dd8190 state=pending>
parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: <AppFuture at 0x14a6f8dd9250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 0e4b1f1db87b84d4beca9862b833eb45
parsl.dataflow.memoization INFO: Task 558 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187
parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dc43e858-cb06-4fc1-8cf1-ef471d59ee5d_isr_947528_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dc43e858-cb06-4fc1-8cf1-ef471d59ee5d_isr_947528_22.stderr
parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558
parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: <AppFuture at 0x14a6f8dd8f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559
parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: <AppFuture at 0x14a6f8dd9a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: <AppFuture at 0x14a6f8dda650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 66602433f59cb70e603c3cf2cf8847e5
parsl.dataflow.memoization INFO: Task 561 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188
parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c1374fb3-3494-4f4c-850b-719ebefd720e_isr_947528_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c1374fb3-3494-4f4c-850b-719ebefd720e_isr_947528_36.stderr
parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561
parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: <AppFuture at 0x14a6f8dd9a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562
parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: <AppFuture at 0x14a6f8ddb550 state=pending>
parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: <AppFuture at 0x14a6f8ddbb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 564 has memoization hash baa45e2ce6041a2a613129d1f0e95b8d
parsl.dataflow.memoization INFO: Task 564 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189
parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/34b8a893-2d25-4cdd-81ac-0e94bd12e5bc_isr_947526_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/34b8a893-2d25-4cdd-81ac-0e94bd12e5bc_isr_947526_59.stderr
parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564
parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: <AppFuture at 0x14a6f8ddbfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565
parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: <AppFuture at 0x14a6f8de0990 state=pending>
parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: <AppFuture at 0x14a6f8de1090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 1f93a72dad14874801ac3ad73cdb72bf
parsl.dataflow.memoization INFO: Task 567 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190
parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9f8579cc-848f-4068-86ab-44c8dae9c955_isr_947420_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9f8579cc-848f-4068-86ab-44c8dae9c955_isr_947420_12.stderr
parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567
parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: <AppFuture at 0x14a6f8de02d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568
parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: <AppFuture at 0x14a6f8de2010 state=pending>
parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: <AppFuture at 0x14a6f8de2510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 721526aa5e820a3ec519fe649987464d
parsl.dataflow.memoization INFO: Task 570 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191
parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5782d2bd-150e-45ed-b08c-c584aa4f5d55_isr_947420_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5782d2bd-150e-45ed-b08c-c584aa4f5d55_isr_947420_16.stderr
parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570
parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: <AppFuture at 0x14a6f8de2bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571
parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: <AppFuture at 0x14a6f8de3510 state=pending>
parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: <AppFuture at 0x14a6f8de3990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 43b4c38753307c9ee76971e004b98322
parsl.dataflow.memoization INFO: Task 573 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192
parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/269d1423-0ac3-4181-a4e3-9a36909ae0d9_isr_947524_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/269d1423-0ac3-4181-a4e3-9a36909ae0d9_isr_947524_29.stderr
parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573
parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: <AppFuture at 0x14a6f8de3d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574
parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: <AppFuture at 0x14a6f8dec950 state=pending>
parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: <AppFuture at 0x14a6f8dece50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 8c914b2424013f0edbb28e552cd79a8d
parsl.dataflow.memoization INFO: Task 576 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193
parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6513be22-6ed4-40f0-a836-8f6dd03a8551_isr_947523_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6513be22-6ed4-40f0-a836-8f6dd03a8551_isr_947523_23.stderr
parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576
parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: <AppFuture at 0x14a6f8dec1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577
parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: <AppFuture at 0x14a6f8dedd50 state=pending>
parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: <AppFuture at 0x14a6f8dee310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 579 has memoization hash b7c6590b8d90f66b5883296ecd6ead7e
parsl.dataflow.memoization INFO: Task 579 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194
parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/35120db0-0364-4cd2-818c-cd24480ee67e_isr_947420_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/35120db0-0364-4cd2-818c-cd24480ee67e_isr_947420_55.stderr
parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579
parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: <AppFuture at 0x14a6f8ded8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580
parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: <AppFuture at 0x14a6f8def4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: <AppFuture at 0x14a6f8def8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 582 has memoization hash f79043ec174c2db6584e4de6fe9e9ad5
parsl.dataflow.memoization INFO: Task 582 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195
parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9005f966-5a16-4228-9c11-9624e504fbd2_isr_947524_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9005f966-5a16-4228-9c11-9624e504fbd2_isr_947524_30.stderr
parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582
parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: <AppFuture at 0x14a6f8defc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583
parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: <AppFuture at 0x14a6f8df4710 state=pending>
parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: <AppFuture at 0x14a6f8df4c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 469c787d240ba34f56dbfa2a6bbf3ca8
parsl.dataflow.memoization INFO: Task 585 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196
parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/ebdd6fda-6535-49cb-93c8-e8946bf05ee0_isr_947523_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/ebdd6fda-6535-49cb-93c8-e8946bf05ee0_isr_947523_1.stderr
parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585
parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: <AppFuture at 0x14a6f8df4050 state=pending>
parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586
parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: <AppFuture at 0x14a6f8df5190 state=pending>
parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: <AppFuture at 0x14a6f8df6050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 588 has memoization hash b7a1bbd096de41295c4c209314fccb4f
parsl.dataflow.memoization INFO: Task 588 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197
parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/99cb83ab-e2eb-417f-bd5d-6641b94e015c_isr_947422_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/99cb83ab-e2eb-417f-bd5d-6641b94e015c_isr_947422_47.stderr
parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588
parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: <AppFuture at 0x14a6f8df6350 state=pending>
parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589
parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: <AppFuture at 0x14a6f8df6f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: <AppFuture at 0x14a6f8df7550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 641490bb1c03fcaea2ac8536929b5670
parsl.dataflow.memoization INFO: Task 591 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198
parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/428cc318-bcd9-44bb-bef7-799f0fde279d_isr_947422_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/428cc318-bcd9-44bb-bef7-799f0fde279d_isr_947422_13.stderr
parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591
parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: <AppFuture at 0x14a6f8df7290 state=pending>
parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592
parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: <AppFuture at 0x14a6f8df7a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: <AppFuture at 0x14a6f8dfc910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 9bb92606b6887b09d2e665db44b7bb7b
parsl.dataflow.memoization INFO: Task 594 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199
parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b1053073-5cf5-4b60-a08f-61415e2c3278_isr_947422_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b1053073-5cf5-4b60-a08f-61415e2c3278_isr_947422_53.stderr
parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594
parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: <AppFuture at 0x14a6f8dfd290 state=pending>
parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595
parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: <AppFuture at 0x14a6f8dfd7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: <AppFuture at 0x14a6f8dfdd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 597 has memoization hash e92b607b40de9335795a61050a1ba347
parsl.dataflow.memoization INFO: Task 597 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200
parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2dce7b54-10d2-43a9-8370-cb4c0b695061_isr_947422_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2dce7b54-10d2-43a9-8370-cb4c0b695061_isr_947422_38.stderr
parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597
parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: <AppFuture at 0x14a6f8dfdad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598
parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: <AppFuture at 0x14a6f8dfee10 state=pending>
parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: <AppFuture at 0x14a6f8dff290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 600 has memoization hash fb3c75f8b9bd8edb742d188ea7e36520
parsl.dataflow.memoization INFO: Task 600 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201
parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/812a58cb-ad4e-4bda-81ab-0c837769aed9_isr_947526_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/812a58cb-ad4e-4bda-81ab-0c837769aed9_isr_947526_27.stderr
parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600
parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: <AppFuture at 0x14a6f8dff810 state=pending>
parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601
parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: <AppFuture at 0x14a6f8dff710 state=pending>
parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0c750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 7141b367461812a0b12b8be4acd09ab2
parsl.dataflow.memoization INFO: Task 603 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202
parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/888a9dc1-b2e8-4597-b25f-61c46853e0fd_isr_947524_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/888a9dc1-b2e8-4597-b25f-61c46853e0fd_isr_947524_62.stderr
parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603
parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0c490 state=pending>
parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604
parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0cc10 state=pending>
parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0db90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 606 has memoization hash c34b1fa7cd96dc93854b76d814559779
parsl.dataflow.memoization INFO: Task 606 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203
parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3008513e-abd5-469a-ab8d-4ec4b855d921_isr_947418_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3008513e-abd5-469a-ab8d-4ec4b855d921_isr_947418_19.stderr
parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606
parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0e950 state=pending>
parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607
parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0e850 state=pending>
parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0f050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 609 has memoization hash d2c187bd5e9d1ffa9006a658f5d6d771
parsl.dataflow.memoization INFO: Task 609 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204
parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/12a48e3a-00e8-4dd6-ab91-bd2b43c77f51_isr_947526_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/12a48e3a-00e8-4dd6-ab91-bd2b43c77f51_isr_947526_62.stderr
parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609
parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0f3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610
parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0f8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: <AppFuture at 0x14a6f8e14590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 61edcc07be2d4913d5c8b7f68dc00c14
parsl.dataflow.memoization INFO: Task 612 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dceac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205
parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/853956f0-e029-44af-b79b-939a5ef4a871_isr_947422_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/853956f0-e029-44af-b79b-939a5ef4a871_isr_947422_39.stderr
parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612
parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: <AppFuture at 0x14a6f8e0e6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613
parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: <AppFuture at 0x14a6f8e15690 state=pending>
parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: <AppFuture at 0x14a6f8e159d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 615 has memoization hash b6830a3e598c47285601b90abed4eb0c
parsl.dataflow.memoization INFO: Task 615 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206
parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5347f695-2223-43d2-a7ce-3565cb6a6faa_isr_947526_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5347f695-2223-43d2-a7ce-3565cb6a6faa_isr_947526_9.stderr
parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615
parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: <AppFuture at 0x14a6f8e16110 state=pending>
parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616
parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: <AppFuture at 0x14a6f8e16290 state=pending>
parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: <AppFuture at 0x14a6f8e16e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 52c7c654421713f5442a573a07dbcd0e
parsl.dataflow.memoization INFO: Task 618 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207
parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0a97e0e4-4525-4341-802c-6343a452b462_isr_947526_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0a97e0e4-4525-4341-802c-6343a452b462_isr_947526_60.stderr
parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618
parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: <AppFuture at 0x14a6f8e16bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619
parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: <AppFuture at 0x14a6f8e17ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: <AppFuture at 0x14a6f8e20310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 621 has memoization hash c2895b64c04911025e79133d52ce910b
parsl.dataflow.memoization INFO: Task 621 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208
parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/03c347ad-d0e2-4d3c-ac18-af3c8e612493_isr_947523_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/03c347ad-d0e2-4d3c-ac18-af3c8e612493_isr_947523_55.stderr
parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621
parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: <AppFuture at 0x14a6f8e17f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622
parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: <AppFuture at 0x14a6f8e20d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: <AppFuture at 0x14a6f8e21610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 59fff7bfb0ef407fc8e67627251fb897
parsl.dataflow.memoization INFO: Task 624 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209
parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d7dc86f2-8e0f-4179-851b-1c8b222f22b8_isr_947420_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d7dc86f2-8e0f-4179-851b-1c8b222f22b8_isr_947420_20.stderr
parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624
parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: <AppFuture at 0x14a6f8e17e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625
parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: <AppFuture at 0x14a6f8e21b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: <AppFuture at 0x14a6f8e22ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 627 has memoization hash e483685af79aa25ef57d7126641bd9c6
parsl.dataflow.memoization INFO: Task 627 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210
parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c31eb02e-4092-49fa-b2b9-1a2d49b75ec0_isr_947422_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c31eb02e-4092-49fa-b2b9-1a2d49b75ec0_isr_947422_49.stderr
parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627
parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: <AppFuture at 0x14a6f8e21e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628
parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: <AppFuture at 0x14a6f8e23910 state=pending>
parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: <AppFuture at 0x14a6f8e23f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 43105f9778ac527c55d6f38fb194de93
parsl.dataflow.memoization INFO: Task 630 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211
parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0287b4ba-df83-40e9-b633-4fee07200799_isr_947523_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0287b4ba-df83-40e9-b633-4fee07200799_isr_947523_58.stderr
parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630
parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: <AppFuture at 0x14a6f8e23bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631
parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: <AppFuture at 0x14a6f8e24750 state=pending>
parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: <AppFuture at 0x14a6f8e25310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 45e688c1338c2c94f2f87db5f7a97e42
parsl.dataflow.memoization INFO: Task 633 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212
parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5ad49684-06d9-4128-a7cc-a92bb3fa458c_isr_947418_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5ad49684-06d9-4128-a7cc-a92bb3fa458c_isr_947418_45.stderr
parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633
parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: <AppFuture at 0x14a6f8e24c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634
parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: <AppFuture at 0x14a6f8e26210 state=pending>
parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: <AppFuture at 0x14a6f8e26790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 992b2bdaa8f7bfe584a22c73e8dd92bb
parsl.dataflow.memoization INFO: Task 636 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213
parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3c4768e4-6523-4b21-9419-68290dac75ce_isr_947420_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3c4768e4-6523-4b21-9419-68290dac75ce_isr_947420_41.stderr
parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636
parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: <AppFuture at 0x14a6f8e26a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637
parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: <AppFuture at 0x14a6f8e27650 state=pending>
parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: <AppFuture at 0x14a6f8e27c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 71aa318501ad6d0b33f5b1d7071e4dc9
parsl.dataflow.memoization INFO: Task 639 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214
parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2e162512-ff1b-4a2a-95d9-1c4d680e5da3_isr_947528_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2e162512-ff1b-4a2a-95d9-1c4d680e5da3_isr_947528_33.stderr
parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639
parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: <AppFuture at 0x14a6f8e27990 state=pending>
parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640
parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: <AppFuture at 0x14a6f8e384d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: <AppFuture at 0x14a6f8e39090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 23445edb42e3cac16f6df240c85494f5
parsl.dataflow.memoization INFO: Task 642 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215
parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce178b88-649e-4714-9b45-4e1f34fc164e_isr_947524_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce178b88-649e-4714-9b45-4e1f34fc164e_isr_947524_49.stderr
parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642
parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: <AppFuture at 0x14a6f8e39210 state=pending>
parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643
parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: <AppFuture at 0x14a6f8e38390 state=pending>
parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: <AppFuture at 0x14a6f8e3a390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 3e363c86babcc9a6c89a9323236dc60f
parsl.dataflow.memoization INFO: Task 645 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216
parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2f878301-e9a1-4b8f-b0cf-280903cc72c8_isr_947422_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2f878301-e9a1-4b8f-b0cf-280903cc72c8_isr_947422_56.stderr
parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645
parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: <AppFuture at 0x14a6f8e39910 state=pending>
parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646
parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: <AppFuture at 0x14a6f8e3b2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: <AppFuture at 0x14a6f8e3b7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 648 has memoization hash e0ddfa8f0191cb8f0f9aa11d9246fee3
parsl.dataflow.memoization INFO: Task 648 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217
parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/47acf2f8-8840-4dde-a4f6-0dec752c4059_isr_947422_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/47acf2f8-8840-4dde-a4f6-0dec752c4059_isr_947422_16.stderr
parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648
parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: <AppFuture at 0x14a6f8e3bd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649
parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: <AppFuture at 0x14a6f8e40850 state=pending>
parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: <AppFuture at 0x14a6f8e40bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 8e218044316eac1815c6d0591c41bd03
parsl.dataflow.memoization INFO: Task 651 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218
parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6165cb49-1271-46b0-a7e6-27c4443cb9c5_isr_947526_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6165cb49-1271-46b0-a7e6-27c4443cb9c5_isr_947526_26.stderr
parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651
parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: <AppFuture at 0x14a6fc6d9250 state=pending>
parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652
parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: <AppFuture at 0x14a6f8e41890 state=pending>
parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: <AppFuture at 0x14a6f8e420d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 20af12715548692cbcaf92432da1e319
parsl.dataflow.memoization INFO: Task 654 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219
parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cb8b6332-7802-46ec-ab7b-a4e58035d7e4_isr_947523_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cb8b6332-7802-46ec-ab7b-a4e58035d7e4_isr_947523_4.stderr
parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654
parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: <AppFuture at 0x14a6f8e42450 state=pending>
parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655
parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: <AppFuture at 0x14a6f8e42d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: <AppFuture at 0x14a6f8e43510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 657 has memoization hash d186cb806b26bb8ae3e7c931bbb78f18
parsl.dataflow.memoization INFO: Task 657 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220
parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/144becd8-5739-4163-a497-a9d15135e45d_isr_947526_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/144becd8-5739-4163-a497-a9d15135e45d_isr_947526_56.stderr
parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657
parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: <AppFuture at 0x14a6f8e427d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658
parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: <AppFuture at 0x14a6f8e43a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: <AppFuture at 0x14a6f8e48890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 660 has memoization hash b0ee03dc495efb6854d61fc793e9f177
parsl.dataflow.memoization INFO: Task 660 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221
parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6ff96fad-8cf2-4f1f-a226-d4b3d29c567a_isr_947526_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6ff96fad-8cf2-4f1f-a226-d4b3d29c567a_isr_947526_8.stderr
parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660
parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: <AppFuture at 0x14a6f8e43c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661
parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: <AppFuture at 0x14a6f8e49550 state=pending>
parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: <AppFuture at 0x14a6f8e49d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 663 has memoization hash b1c247fcb130d7b46d96dda5f509e4df
parsl.dataflow.memoization INFO: Task 663 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222
parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/51371269-7373-482b-acab-f14804720e35_isr_947422_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/51371269-7373-482b-acab-f14804720e35_isr_947422_31.stderr
parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663
parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: <AppFuture at 0x14a6f8e492d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664
parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: <AppFuture at 0x14a6f8e4a810 state=pending>
parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: <AppFuture at 0x14a6f8e4b150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 666 has memoization hash b7d5e305971201bb32be65fa002f22e7
parsl.dataflow.memoization INFO: Task 666 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223
parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/51d42e7d-a1e7-4581-b813-4f180f5d0388_isr_947420_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/51d42e7d-a1e7-4581-b813-4f180f5d0388_isr_947420_28.stderr
parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666
parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: <AppFuture at 0x14a6f8e4bb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667
parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: <AppFuture at 0x14a6f8e4b650 state=pending>
parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: <AppFuture at 0x14a6f8e585d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 669 has memoization hash dc23e9f510af184192dd0e56e3e8238f
parsl.dataflow.memoization INFO: Task 669 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224
parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/28a028dc-01ec-4f2d-9b90-475bd8eb563f_isr_947420_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/28a028dc-01ec-4f2d-9b90-475bd8eb563f_isr_947420_35.stderr
parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669
parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: <AppFuture at 0x14a6f8e58750 state=pending>
parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670
parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: <AppFuture at 0x14a6f8e58e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: <AppFuture at 0x14a6f8e59990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 672 has memoization hash b95bce87394a896fbc19df12d9d230dd
parsl.dataflow.memoization INFO: Task 672 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225
parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b38a1018-d7c6-41d6-8c85-96f6d469410c_isr_947422_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b38a1018-d7c6-41d6-8c85-96f6d469410c_isr_947422_29.stderr
parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672
parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: <AppFuture at 0x14a6f8e59d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673
parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: <AppFuture at 0x14a6f8e5a890 state=pending>
parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: <AppFuture at 0x14a6f8e5af10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 5edd5f969fa23780d3737d17e39d1d20
parsl.dataflow.memoization INFO: Task 675 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226
parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d0f28290-0b57-4f13-b110-82f5fd551725_isr_947418_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d0f28290-0b57-4f13-b110-82f5fd551725_isr_947418_55.stderr
parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675
parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: <AppFuture at 0x14a6f8e5b090 state=pending>
parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676
parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: <AppFuture at 0x14a6f8e5bd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: <AppFuture at 0x14a6f8e60350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 678 has memoization hash f56c955680c2e28d78f6e6fe7624e608
parsl.dataflow.memoization INFO: Task 678 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227
parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c4dfdd93-19e8-4448-995c-11a369974d8c_isr_947422_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c4dfdd93-19e8-4448-995c-11a369974d8c_isr_947422_55.stderr
parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678
parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: <AppFuture at 0x14a6f8e61050 state=pending>
parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679
parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: <AppFuture at 0x14a6f8e60ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: <AppFuture at 0x14a6f8e61750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 681 has memoization hash fbe8754d269e911417aeb9f242ced1ef
parsl.dataflow.memoization INFO: Task 681 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228
parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bcf96450-37b5-46d6-befd-74ee72cf7924_isr_947528_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bcf96450-37b5-46d6-befd-74ee72cf7924_isr_947528_62.stderr
parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681
parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: <AppFuture at 0x14a6f8e626d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682
parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: <AppFuture at 0x14a6f8e62090 state=pending>
parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: <AppFuture at 0x14a6f8e62b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 462d4b6bb2390c7259fc36890242454a
parsl.dataflow.memoization INFO: Task 684 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229
parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b737d956-790c-40ef-8a5b-97d73f16486f_isr_947418_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b737d956-790c-40ef-8a5b-97d73f16486f_isr_947418_31.stderr
parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684
parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: <AppFuture at 0x14a6f8e62810 state=pending>
parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685
parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: <AppFuture at 0x14a6f8e639d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: <AppFuture at 0x14a6f8e63f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 687 has memoization hash b2dfdc898ea7714967e21564a273ebff
parsl.dataflow.memoization INFO: Task 687 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230
parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/763d8556-4a05-4045-835f-60ec0c718de1_isr_947526_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/763d8556-4a05-4045-835f-60ec0c718de1_isr_947526_31.stderr
parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687
parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: <AppFuture at 0x14a6f8e5b910 state=pending>
parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688
parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6ced0 state=pending>
parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6d410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 0d254f20eddedfadaed01d1017c5347e
parsl.dataflow.memoization INFO: Task 690 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231
parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ac75fc94-67ad-4f36-b9eb-f28b9533566a_isr_947422_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ac75fc94-67ad-4f36-b9eb-f28b9533566a_isr_947422_62.stderr
parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690
parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6cad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691
parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6e550 state=pending>
parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6e950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 693 has memoization hash e51ee7de9300b01b534dd8fb7a8867d2
parsl.dataflow.memoization INFO: Task 693 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232
parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/075952bc-6a80-4c63-a5e9-acdbac292d31_isr_947422_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/075952bc-6a80-4c63-a5e9-acdbac292d31_isr_947422_4.stderr
parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693
parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6ec50 state=pending>
parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694
parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6f990 state=pending>
parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6fdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 696393f570a0bf3aa609c2d6e46f0f64
parsl.dataflow.memoization INFO: Task 696 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233
parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1ab2b1d6-3583-499a-b0af-02646c0b4388_isr_947526_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1ab2b1d6-3583-499a-b0af-02646c0b4388_isr_947526_20.stderr
parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696
parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: <AppFuture at 0x14a6f8e6e310 state=pending>
parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697
parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7cc50 state=pending>
parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7d250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 699 has memoization hash f6ab75ab3c8a3ef00c8e459677ff4423
parsl.dataflow.memoization INFO: Task 699 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234
parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47f6608e-1190-4440-8ebb-ce560a1e0e52_isr_947420_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47f6608e-1190-4440-8ebb-ce560a1e0e52_isr_947420_1.stderr
parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699
parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7ced0 state=pending>
parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700
parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7e190 state=pending>
parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7e690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 6b2bbd9881ca30efa125616988781d95
parsl.dataflow.memoization INFO: Task 702 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235
parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f57f57b6-b782-4b84-97cd-ff27572bf432_isr_947420_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f57f57b6-b782-4b84-97cd-ff27572bf432_isr_947420_33.stderr
parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702
parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7ea10 state=pending>
parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703
parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7f810 state=pending>
parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7fc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 508266edc39f99f4ca163795b9623a4b
parsl.dataflow.memoization INFO: Task 705 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236
parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9424d2f-107c-40dc-9aae-e747d9032972_isr_947422_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9424d2f-107c-40dc-9aae-e747d9032972_isr_947422_32.stderr
parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705
parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: <AppFuture at 0x14a6f8c7f8d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706
parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: <AppFuture at 0x14a6f8c88250 state=pending>
parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: <AppFuture at 0x14a6f8c88fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 708 has memoization hash b5248263c1062636121a46dee75f6697
parsl.dataflow.memoization INFO: Task 708 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237
parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/46eb6ebd-503b-435d-a8f3-7ba900c2f5b3_isr_947526_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/46eb6ebd-503b-435d-a8f3-7ba900c2f5b3_isr_947526_5.stderr
parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708
parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: <AppFuture at 0x14a6f8c89350 state=pending>
parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709
parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: <AppFuture at 0x14a6f8c89e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: <AppFuture at 0x14a6f8c8a450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 711 has memoization hash d53d5b37f8af6d6d0b45896b26703770
parsl.dataflow.memoization INFO: Task 711 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238
parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/71a06537-8313-4031-9a29-510210d54f0f_isr_947523_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/71a06537-8313-4031-9a29-510210d54f0f_isr_947523_44.stderr
parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711
parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: <AppFuture at 0x14a6f8c8a590 state=pending>
parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712
parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: <AppFuture at 0x14a6f8c8b050 state=pending>
parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: <AppFuture at 0x14a6f8c8b7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 714 has memoization hash dff0795067d75f0c4b19b28517e8abd9
parsl.dataflow.memoization INFO: Task 714 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239
parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/d7d27b5a-9626-4782-a8c9-9670aa65748e_isr_947523_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/d7d27b5a-9626-4782-a8c9-9670aa65748e_isr_947523_62.stderr
parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714
parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: <AppFuture at 0x14a6f8c8bad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715
parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: <AppFuture at 0x14a6f8c903d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: <AppFuture at 0x14a6f8c90c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 7b4ca368e2c47d4d07addfac6490c44b
parsl.dataflow.memoization INFO: Task 717 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240
parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/43adf152-7201-4d10-ac4e-6b91eb464880_isr_947420_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/43adf152-7201-4d10-ac4e-6b91eb464880_isr_947420_15.stderr
parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717
parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: <AppFuture at 0x14a6f8c90210 state=pending>
parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718
parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: <AppFuture at 0x14a6f8c91a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: <AppFuture at 0x14a6f8c92010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 3d1d3ef1891a9a5836a75b49bf2512e5
parsl.dataflow.memoization INFO: Task 720 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241
parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/991b0057-89e2-4f20-a4d8-e51a998457ec_isr_947422_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/991b0057-89e2-4f20-a4d8-e51a998457ec_isr_947422_48.stderr
parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720
parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: <AppFuture at 0x14a6f8c91390 state=pending>
parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721
parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: <AppFuture at 0x14a6f8c92e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: <AppFuture at 0x14a6f8c93390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 198cc807805b1e5ae08d94d631a9c740
parsl.dataflow.memoization INFO: Task 723 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242
parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f2e3aa78-c652-4708-9a49-0ace4d7078af_isr_947526_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f2e3aa78-c652-4708-9a49-0ace4d7078af_isr_947526_53.stderr
parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723
parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: <AppFuture at 0x14a6f8c930d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724
parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca0150 state=pending>
parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca0850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 7754fc09147f5d55bacdcf1fc311a088
parsl.dataflow.memoization INFO: Task 726 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243
parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/94737fc6-01d8-4d65-9038-1aecd31305fa_isr_947418_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/94737fc6-01d8-4d65-9038-1aecd31305fa_isr_947418_48.stderr
parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726
parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca0590 state=pending>
parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727
parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca1650 state=pending>
parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca1bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 8272d6f7898574a7eb1b0bc1f00e89eb
parsl.dataflow.memoization INFO: Task 729 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244
parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4b69b6a4-25f2-482c-a405-e1042b7e2ff1_isr_947420_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4b69b6a4-25f2-482c-a405-e1042b7e2ff1_isr_947420_19.stderr
parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729
parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: <AppFuture at 0x14a6f8e58fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730
parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca2850 state=pending>
parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca2f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 732 has memoization hash d71120519c48a465075c48f213239962
parsl.dataflow.memoization INFO: Task 732 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245
parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/156cbabe-87c7-4aa0-bb65-66f0eef27c64_isr_947524_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/156cbabe-87c7-4aa0-bb65-66f0eef27c64_isr_947524_43.stderr
parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732
parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca3290 state=pending>
parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733
parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca3bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca4450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 735 has memoization hash 66db025262529447ddf6cf0e1999c5ef
parsl.dataflow.memoization INFO: Task 735 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246
parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/79891680-d9ea-4e28-870c-6375b4cfc02f_isr_947528_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/79891680-d9ea-4e28-870c-6375b4cfc02f_isr_947528_27.stderr
parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735
parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca3750 state=pending>
parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736
parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca4bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca5890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 94bd4c0551a756cc70f3e904f09770e0
parsl.dataflow.memoization INFO: Task 738 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247
parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0f762b41-cc37-46f3-b657-add23d7c99fe_isr_947422_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0f762b41-cc37-46f3-b657-add23d7c99fe_isr_947422_28.stderr
parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738
parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca4f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739
parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca6750 state=pending>
parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca6cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 11fc42d3309f102a515642e1264ffec6
parsl.dataflow.memoization INFO: Task 741 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248
parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/412090d0-18be-46df-a163-8232d842da9e_isr_947524_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/412090d0-18be-46df-a163-8232d842da9e_isr_947524_7.stderr
parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741
parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca7510 state=pending>
parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742
parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca74d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: <AppFuture at 0x14a6f8cac110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 744 has memoization hash f290585e26af439c091d0d3445ccd6ce
parsl.dataflow.memoization INFO: Task 744 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249
parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0a05a638-ae4c-4dc5-b606-b6499d9bc393_isr_947524_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0a05a638-ae4c-4dc5-b606-b6499d9bc393_isr_947524_45.stderr
parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744
parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: <AppFuture at 0x14a6f8ca7d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745
parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: <AppFuture at 0x14a6f8cacb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: <AppFuture at 0x14a6f8cad450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 183d4b0ab5c745d74039f3a7c565cf77
parsl.dataflow.memoization INFO: Task 747 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250
parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/301c801c-8c29-4ee9-bf3d-a6f2dab45a70_isr_947523_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/301c801c-8c29-4ee9-bf3d-a6f2dab45a70_isr_947523_38.stderr
parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747
parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: <AppFuture at 0x14a6f8cad5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748
parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: <AppFuture at 0x14a6f8cae350 state=pending>
parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: <AppFuture at 0x14a6f8cae950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 750 has memoization hash f06c221734fd9e6e45aa2379a0232e2b
parsl.dataflow.memoization INFO: Task 750 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251
parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/180fc813-90c2-41ec-ac13-4d3ae16aa25f_isr_947422_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/180fc813-90c2-41ec-ac13-4d3ae16aa25f_isr_947422_22.stderr
parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750
parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: <AppFuture at 0x14a6f8caea50 state=pending>
parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751
parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: <AppFuture at 0x14a6f8caf850 state=pending>
parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: <AppFuture at 0x14a6f8cafd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 5fa7447229b08ff35f2299b5cdabf58e
parsl.dataflow.memoization INFO: Task 753 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252
parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7836fd93-b6cf-438d-a30e-bbf41e6d355d_isr_947523_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7836fd93-b6cf-438d-a30e-bbf41e6d355d_isr_947523_17.stderr
parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753
parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: <AppFuture at 0x14a6f8caf990 state=pending>
parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754
parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbc650 state=pending>
parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbd150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 1bac6392a4e780bc4b21970c51f72474
parsl.dataflow.memoization INFO: Task 756 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253
parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b9e0f4f8-e3da-487f-98ce-604b0f8f4dc5_isr_947523_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b9e0f4f8-e3da-487f-98ce-604b0f8f4dc5_isr_947523_40.stderr
parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756
parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbc510 state=pending>
parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757
parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbe010 state=pending>
parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbe5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 085505e044c7577a28594d468217ac36
parsl.dataflow.memoization INFO: Task 759 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254
parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bea57d7-e3dc-424c-acb1-24877ca19f76_isr_947420_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bea57d7-e3dc-424c-acb1-24877ca19f76_isr_947420_59.stderr
parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759
parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbf450 state=pending>
parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760
parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbec90 state=pending>
parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbf990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 85505869a04d0cc4c94f28c4fc9ee210
parsl.dataflow.memoization INFO: Task 762 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255
parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f161eb3c-3d1a-45f0-bb3c-d8be45f14274_isr_947528_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f161eb3c-3d1a-45f0-bb3c-d8be45f14274_isr_947528_60.stderr
parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762
parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: <AppFuture at 0x14a6f8cbfed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763
parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccc810 state=pending>
parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccce10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 1e6ed2156808fb2ef452cfdc5b51bfdb
parsl.dataflow.memoization INFO: Task 765 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256
parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/607ebb04-74fc-4f24-869c-be5bfa2c5915_isr_947528_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/607ebb04-74fc-4f24-869c-be5bfa2c5915_isr_947528_14.stderr
parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765
parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccd410 state=pending>
parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766
parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccd710 state=pending>
parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: <AppFuture at 0x14a6f8cce210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 8bba5a123f942d3e6501a49fff7bbf22
parsl.dataflow.memoization INFO: Task 768 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257
parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/29b568c4-11c4-49ba-8654-2115fbefa49a_isr_947524_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/29b568c4-11c4-49ba-8654-2115fbefa49a_isr_947524_42.stderr
parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768
parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: <AppFuture at 0x14a6f8cce590 state=pending>
parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769
parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccf150 state=pending>
parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccf690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 9317b4a700b664fa22c61b9edf9864da
parsl.dataflow.memoization INFO: Task 771 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258
parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/b465e888-84ec-481a-a021-1cf38cd4d17b_isr_947420_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/b465e888-84ec-481a-a021-1cf38cd4d17b_isr_947420_39.stderr
parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771
parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccf810 state=pending>
parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772
parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: <AppFuture at 0x14a6f8ccfcd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: <AppFuture at 0x14a6f8cd8ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 52bc2ad880aa2ce8fe63bccd9e0b5a0b
parsl.dataflow.memoization INFO: Task 774 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259
parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/476299b0-c08c-464c-b2d7-e569ce4bc021_isr_947528_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/476299b0-c08c-464c-b2d7-e569ce4bc021_isr_947528_58.stderr
parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774
parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: <AppFuture at 0x14a6f8cd83d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775
parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: <AppFuture at 0x14a6f8cd9350 state=pending>
parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: <AppFuture at 0x14a6f8cd9ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 777 has memoization hash b4a4649c5df9edc2056229eb92aca335
parsl.dataflow.memoization INFO: Task 777 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260
parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2bfab70a-6240-4540-8571-76ec8a20288c_isr_947523_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2bfab70a-6240-4540-8571-76ec8a20288c_isr_947523_61.stderr
parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777
parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: <AppFuture at 0x14a6f8cda050 state=pending>
parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778
parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: <AppFuture at 0x14a6f8cdafd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: <AppFuture at 0x14a6f8cdb2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 3307fd58b7b0574b6fe90424302f294d
parsl.dataflow.memoization INFO: Task 780 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261
parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/134b8e20-ae21-4805-8772-cac90daa9276_isr_947422_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/134b8e20-ae21-4805-8772-cac90daa9276_isr_947422_44.stderr
parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780
parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: <AppFuture at 0x14a6f8cdb5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781
parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: <AppFuture at 0x14a6f8cdb7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce0710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 783 has memoization hash ba494717c3189ccafab75c2a9e6ab8ed
parsl.dataflow.memoization INFO: Task 783 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262
parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/89d89f54-ae01-42d6-b2be-d0fbe877dbea_isr_947523_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/89d89f54-ae01-42d6-b2be-d0fbe877dbea_isr_947523_49.stderr
parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783
parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce1050 state=pending>
parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784
parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce15d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce1b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 5fcd80facd22767af857986afd471648
parsl.dataflow.memoization INFO: Task 786 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263
parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5d1d7ba8-0715-40c1-b0c8-e5f2406facd3_isr_947524_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5d1d7ba8-0715-40c1-b0c8-e5f2406facd3_isr_947524_16.stderr
parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786
parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce17d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787
parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce25d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce3050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 9d2ee007bbdfbae2a6408dc4d7590743
parsl.dataflow.memoization INFO: Task 789 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264
parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/390c5d23-6fb0-4c98-845e-8042df6b6e27_isr_947422_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/390c5d23-6fb0-4c98-845e-8042df6b6e27_isr_947422_24.stderr
parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789
parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce3190 state=pending>
parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790
parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce3a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce8510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 792 has memoization hash d2887590a292bc3d2f14c350a18b67cc
parsl.dataflow.memoization INFO: Task 792 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265
parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f8d8d229-bc52-467a-86fa-8b6cbd4d5ef4_isr_947422_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f8d8d229-bc52-467a-86fa-8b6cbd4d5ef4_isr_947422_19.stderr
parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792
parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce37d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793
parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce8e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce9890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 25b612b0302d704324ca72e7b4918378
parsl.dataflow.memoization INFO: Task 795 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266
parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9002e6c0-24f9-4a1a-ad26-407bb72e0235_isr_947420_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9002e6c0-24f9-4a1a-ad26-407bb72e0235_isr_947420_14.stderr
parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795
parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: <AppFuture at 0x14a6f8cea710 state=pending>
parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796
parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: <AppFuture at 0x14a6f8cea810 state=pending>
parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: <AppFuture at 0x14a6f8cead10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 993204158e2beff107612086a8f39774
parsl.dataflow.memoization INFO: Task 798 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267
parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6c04b92e-7a9f-4932-955f-cb6902014982_isr_947524_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6c04b92e-7a9f-4932-955f-cb6902014982_isr_947524_55.stderr
parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798
parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: <AppFuture at 0x14a6f8cea350 state=pending>
parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799
parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: <AppFuture at 0x14a6f8ceb1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf4050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 537dee68b50e8cc3fd28399a8c367626
parsl.dataflow.memoization INFO: Task 801 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268
parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e58e469-e681-4adc-850d-26c38accfc64_isr_947524_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e58e469-e681-4adc-850d-26c38accfc64_isr_947524_1.stderr
parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801
parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: <AppFuture at 0x14a6f8cebe50 state=pending>
parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802
parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf48d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf5490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 804 has memoization hash fcad45cd2ad4a6a902b24e0dfd9b0728
parsl.dataflow.memoization INFO: Task 804 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269
parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c472aede-cfaa-4cce-92c7-f23d43adace6_isr_947528_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c472aede-cfaa-4cce-92c7-f23d43adace6_isr_947528_40.stderr
parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804
parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf5810 state=pending>
parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805
parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf64d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf6990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 807 has memoization hash b49a51a135ddbc85a80c87ef1c4cde54
parsl.dataflow.memoization INFO: Task 807 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270
parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fdd87d7b-e4f3-4341-a456-75fe39ced94d_isr_947523_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fdd87d7b-e4f3-4341-a456-75fe39ced94d_isr_947523_33.stderr
parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807
parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf6c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808
parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf75d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf7dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 147c8365a470fe301345f7aa767d0331
parsl.dataflow.memoization INFO: Task 810 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271
parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3a6ba6e9-0453-4291-8285-f771668f5f23_isr_947418_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3a6ba6e9-0453-4291-8285-f771668f5f23_isr_947418_35.stderr
parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810
parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: <AppFuture at 0x14a6f8cf7090 state=pending>
parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811
parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: <AppFuture at 0x14a6f8d04810 state=pending>
parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: <AppFuture at 0x14a6f8d051d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 84e0d2378316c884da74a40f93df418d
parsl.dataflow.memoization INFO: Task 813 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272
parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e280fcc3-b7f2-42f0-ad4d-afe98fa93c3a_isr_947418_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e280fcc3-b7f2-42f0-ad4d-afe98fa93c3a_isr_947418_62.stderr
parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813
parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: <AppFuture at 0x14a6f8d04f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814
parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: <AppFuture at 0x14a6f8d06150 state=pending>
parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: <AppFuture at 0x14a6f8d06610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 4e56e75b4195ba15bf6e34367a530349
parsl.dataflow.memoization INFO: Task 816 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273
parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0297da2-2c23-4093-911e-187021eb8f2b_isr_947524_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0297da2-2c23-4093-911e-187021eb8f2b_isr_947524_48.stderr
parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816
parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: <AppFuture at 0x14a6f8d04a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817
parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: <AppFuture at 0x14a6f8d07150 state=pending>
parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: <AppFuture at 0x14a6f8d07a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 51ce13e4101f7d26ad8ea8cad87622e4
parsl.dataflow.memoization INFO: Task 819 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274
parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4898456f-9696-4ade-9ab1-fd0f2f735d0d_isr_947526_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4898456f-9696-4ade-9ab1-fd0f2f735d0d_isr_947526_15.stderr
parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819
parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: <AppFuture at 0x14a6f8d07e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820
parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0c490 state=pending>
parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0ce10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 822 has memoization hash ecb89e14a70d60c41fc5a58a69c8ff94
parsl.dataflow.memoization INFO: Task 822 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275
parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/851612ad-54c3-4d74-a02f-e2513bc5484e_isr_947528_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/851612ad-54c3-4d74-a02f-e2513bc5484e_isr_947528_37.stderr
parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822
parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0d7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823
parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0d810 state=pending>
parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0e1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 3d5c1a8a827f6075df1771964383413c
parsl.dataflow.memoization INFO: Task 825 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d089a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276
parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/df7e5251-fd39-45e9-899a-4a29c6d924aa_isr_947528_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/df7e5251-fd39-45e9-899a-4a29c6d924aa_isr_947528_56.stderr
parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825
parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0e550 state=pending>
parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826
parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0f050 state=pending>
parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0f650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 3af8338daaed2884f5b90bcd7dcf78ec
parsl.dataflow.memoization INFO: Task 828 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277
parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/87b01ebc-c2c8-42ab-8fb2-b0ce87063ebe_isr_947420_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/87b01ebc-c2c8-42ab-8fb2-b0ce87063ebe_isr_947420_38.stderr
parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828
parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0f9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829
parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: <AppFuture at 0x14a6f8d0fdd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: <AppFuture at 0x14a6f8d14a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 831 has memoization hash bb8e751ab08305bad41e0f53d1f345b8
parsl.dataflow.memoization INFO: Task 831 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278
parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/836bdf8d-1719-4816-b0c5-6548241a9377_isr_947418_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/836bdf8d-1719-4816-b0c5-6548241a9377_isr_947418_52.stderr
parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831
parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: <AppFuture at 0x14a6f8d14bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832
parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: <AppFuture at 0x14a6f8d158d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: <AppFuture at 0x14a6f8d15f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 834 has memoization hash caff1112f0b7132c845aa2b521b61c50
parsl.dataflow.memoization INFO: Task 834 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279
parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e6e2e012-352a-4fe4-a3fe-98ebe3213504_isr_947418_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e6e2e012-352a-4fe4-a3fe-98ebe3213504_isr_947418_57.stderr
parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834
parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: <AppFuture at 0x14a6f8d15750 state=pending>
parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835
parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: <AppFuture at 0x14a6f8d16f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: <AppFuture at 0x14a6f8d17450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 00ff97e7173ef6e13b4d4949000be61c
parsl.dataflow.memoization INFO: Task 837 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280
parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8e735054-31cf-4dec-875d-b827213f64fd_isr_947420_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8e735054-31cf-4dec-875d-b827213f64fd_isr_947420_31.stderr
parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837
parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: <AppFuture at 0x14a6f8d17790 state=pending>
parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838
parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: <AppFuture at 0x14a6f8d17950 state=pending>
parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: <AppFuture at 0x14a6f8d20850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 840 has memoization hash fa79203abfcb093eb473105be6431fe6
parsl.dataflow.memoization INFO: Task 840 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281
parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bc44b7f8-9b2b-403a-bb2a-0ea3da87b2a1_isr_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bc44b7f8-9b2b-403a-bb2a-0ea3da87b2a1_isr_947526_61.stderr
parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840
parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: <AppFuture at 0x14a6f8d20590 state=pending>
parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841
parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: <AppFuture at 0x14a6f8d21250 state=pending>
parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: <AppFuture at 0x14a6f8d21b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 843 has memoization hash ccf17d9b7f19f3a3ff99c1f9e58d7eaf
parsl.dataflow.memoization INFO: Task 843 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282
parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b0678ad6-eef2-490d-9aad-05078fa32c13_isr_947523_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b0678ad6-eef2-490d-9aad-05078fa32c13_isr_947523_5.stderr
parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843
parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: <AppFuture at 0x14a6f8d21d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844
parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: <AppFuture at 0x14a6f8d22450 state=pending>
parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: <AppFuture at 0x14a6f8d23010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 846 has memoization hash ae89a3ab6dd04c5d8bec9c9ec6a1433a
parsl.dataflow.memoization INFO: Task 846 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283
parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ff138c26-ec66-4994-88ba-25285c3bc7c9_isr_947524_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ff138c26-ec66-4994-88ba-25285c3bc7c9_isr_947524_4.stderr
parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846
parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: <AppFuture at 0x14a6f8d22d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847
parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: <AppFuture at 0x14a6f8d23550 state=pending>
parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2c490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 9c9dbdf4037024ac6f1ace4e4de8b421
parsl.dataflow.memoization INFO: Task 849 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284
parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1264088f-ec8a-4b1e-bc77-20874890dbc3_isr_947523_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1264088f-ec8a-4b1e-bc77-20874890dbc3_isr_947523_41.stderr
parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849
parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: <AppFuture at 0x14a6f8d22410 state=pending>
parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850
parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2cd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2d810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 07be38c7454ba8e212ad1996608c407e
parsl.dataflow.memoization INFO: Task 852 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285
parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6025c777-f1d6-46f9-a822-38858f9d9cfe_isr_947524_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6025c777-f1d6-46f9-a822-38858f9d9cfe_isr_947524_51.stderr
parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852
parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2d950 state=pending>
parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853
parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2e690 state=pending>
parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2ec90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 855 has memoization hash ae0ee6d31e547824875015dc0a2834e5
parsl.dataflow.memoization INFO: Task 855 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286
parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/849467f6-7cf7-4751-b373-c0ce8abbda38_isr_947523_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/849467f6-7cf7-4751-b373-c0ce8abbda38_isr_947523_53.stderr
parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855
parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2e4d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856
parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2f490 state=pending>
parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2fe50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 31e68444690e5e6f03b01148e745fcee
parsl.dataflow.memoization INFO: Task 858 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287
parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a174c0e-0d5d-45cd-971d-4405713ea66c_isr_947422_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a174c0e-0d5d-45cd-971d-4405713ea66c_isr_947422_51.stderr
parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858
parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2f3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859
parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: <AppFuture at 0x14a6f8d38f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: <AppFuture at 0x14a6f8d39450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 861 has memoization hash a36aafc010a96881b887744563eb0d22
parsl.dataflow.memoization INFO: Task 861 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288
parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c6bd6c40-eb1b-4d6e-914f-71cd49afc967_isr_947420_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c6bd6c40-eb1b-4d6e-914f-71cd49afc967_isr_947420_18.stderr
parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861
parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: <AppFuture at 0x14a6f8d39d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862
parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: <AppFuture at 0x14a6f8d39950 state=pending>
parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: <AppFuture at 0x14a6f8d3a890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 75cfe17e12ddacce3461e08ba9bc9fd6
parsl.dataflow.memoization INFO: Task 864 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289
parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8ad6c19b-49cb-4562-883b-2c44040f2a0c_isr_947526_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8ad6c19b-49cb-4562-883b-2c44040f2a0c_isr_947526_16.stderr
parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864
parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: <AppFuture at 0x14a6f8d3a5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865
parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: <AppFuture at 0x14a6f8d3b150 state=pending>
parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: <AppFuture at 0x14a6f8d3bcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 867 has memoization hash c339cd816e4540624e228ef75dcd6dc0
parsl.dataflow.memoization INFO: Task 867 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290
parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/354f8e7e-cece-4ad7-8022-50e41e90be85_isr_947526_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/354f8e7e-cece-4ad7-8022-50e41e90be85_isr_947526_10.stderr
parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867
parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2cb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868
parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: <AppFuture at 0x14a6f8d446d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: <AppFuture at 0x14a6f8d45150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 870 has memoization hash e83c67758c1623afaa4ec59ae2517bbc
parsl.dataflow.memoization INFO: Task 870 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291
parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/29dddbfb-b367-4307-a69b-c8d4f7425d24_isr_947420_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/29dddbfb-b367-4307-a69b-c8d4f7425d24_isr_947420_32.stderr
parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870
parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2cdd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871
parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: <AppFuture at 0x14a6f8d45950 state=pending>
parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: <AppFuture at 0x14a6f8d46510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 4cdf96d875f7cccad9b76252f025bf30
parsl.dataflow.memoization INFO: Task 873 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292
parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f92ba361-b3bd-4c65-acbf-f49f86a8f210_isr_947528_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f92ba361-b3bd-4c65-acbf-f49f86a8f210_isr_947528_11.stderr
parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873
parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: <AppFuture at 0x14a6f8d45a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874
parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: <AppFuture at 0x14a6f8d47450 state=pending>
parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: <AppFuture at 0x14a6f8d47a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 876 has memoization hash b42c1fadab3e4487157d10723dd0729a
parsl.dataflow.memoization INFO: Task 876 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293
parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c5a2d528-fbf9-4701-8c5c-b06e386b5d4b_isr_947418_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c5a2d528-fbf9-4701-8c5c-b06e386b5d4b_isr_947418_33.stderr
parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876
parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: <AppFuture at 0x14a6f8d47cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877
parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: <AppFuture at 0x14a6f8d50910 state=pending>
parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: <AppFuture at 0x14a6f8d50e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 6542a3c74f2431abf97905de3320b3e5
parsl.dataflow.memoization INFO: Task 879 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294
parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/7a714680-a13e-4d8c-8f9e-6c08174fa906_isr_947524_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/7a714680-a13e-4d8c-8f9e-6c08174fa906_isr_947524_18.stderr
parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879
parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: <AppFuture at 0x14a6f8d50390 state=pending>
parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880
parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: <AppFuture at 0x14a6f8d51c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: <AppFuture at 0x14a6f8d52190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 99985cafa158e0d7c350684a9365dd87
parsl.dataflow.memoization INFO: Task 882 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295
parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/a2a9cc77-8ed8-4e17-99f7-653d6cba04a6_isr_947528_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/a2a9cc77-8ed8-4e17-99f7-653d6cba04a6_isr_947528_18.stderr
parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882
parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: <AppFuture at 0x14a6f8d51650 state=pending>
parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883
parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: <AppFuture at 0x14a6f8d52bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: <AppFuture at 0x14a6f8d53550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 885 has memoization hash d59f67dbda6c74c7b67c4a96cf3c3cf2
parsl.dataflow.memoization INFO: Task 885 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296
parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6bf623b7-72cb-4dd4-b5f4-cd8f5c75e6cf_isr_947422_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6bf623b7-72cb-4dd4-b5f4-cd8f5c75e6cf_isr_947422_17.stderr
parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885
parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: <AppFuture at 0x14a6f8d52a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886
parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5c610 state=pending>
parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5ca10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 888 has memoization hash ecb810772c1e5e267cbd808a2c75f50b
parsl.dataflow.memoization INFO: Task 888 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297
parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4fce6d0d-ff21-4ce0-859b-470b5bea8e82_isr_947526_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4fce6d0d-ff21-4ce0-859b-470b5bea8e82_isr_947526_33.stderr
parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888
parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: <AppFuture at 0x14a6f8d45710 state=pending>
parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889
parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5d390 state=pending>
parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5de10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 132fc0086eb1668ff3bd305ca4d84e42
parsl.dataflow.memoization INFO: Task 891 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298
parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/d285b61f-5ec0-416e-83c2-48eaf75989f7_isr_947422_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/d285b61f-5ec0-416e-83c2-48eaf75989f7_isr_947422_14.stderr
parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891
parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5d250 state=pending>
parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892
parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5e710 state=pending>
parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5f250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 58d818f1f2bba9a81176f6c571438594
parsl.dataflow.memoization INFO: Task 894 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299
parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11603f74-603f-46ef-99f2-35cffd3d5e50_isr_947528_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11603f74-603f-46ef-99f2-35cffd3d5e50_isr_947528_4.stderr
parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894
parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5f810 state=pending>
parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895
parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: <AppFuture at 0x14a6f8ce80d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: <AppFuture at 0x14a6f8d68690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 897 has memoization hash df031e179c1b97a4bb422ec705eeb0c6
parsl.dataflow.memoization INFO: Task 897 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300
parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/867b1965-a335-4f22-a3f5-292835011cb0_isr_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/867b1965-a335-4f22-a3f5-292835011cb0_isr_947526_2.stderr
parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897
parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: <AppFuture at 0x14a6f8d687d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898
parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: <AppFuture at 0x14a6f8d68e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: <AppFuture at 0x14a6f8d69a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 900 has memoization hash 5c24c6b0573011462849cafa781c6eb3
parsl.dataflow.memoization INFO: Task 900 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301
parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a8af7d8-0b72-4f2c-b9f0-5c0a7643dc72_isr_947422_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a8af7d8-0b72-4f2c-b9f0-5c0a7643dc72_isr_947422_58.stderr
parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900
parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: <AppFuture at 0x14a6f8d69b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901
parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6a9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6af10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 903 has memoization hash d9eb30509d7fce294e09d2ebcf2fd4ab
parsl.dataflow.memoization INFO: Task 903 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302
parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2529e872-deae-4647-bc18-dbc2736a23fa_isr_947422_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2529e872-deae-4647-bc18-dbc2736a23fa_isr_947422_41.stderr
parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903
parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6ab90 state=pending>
parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904
parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6b750 state=pending>
parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6c2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 906 has memoization hash d0a444f6ef756fb71d33e0e5773f2c65
parsl.dataflow.memoization INFO: Task 906 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303
parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5695aa7f-c6df-4f95-8fab-feb095a579a5_isr_947523_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5695aa7f-c6df-4f95-8fab-feb095a579a5_isr_947523_15.stderr
parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906
parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6b710 state=pending>
parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907
parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6c7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6d710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 21149f887c7ea003c72d83ad16536ba8
parsl.dataflow.memoization INFO: Task 909 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304
parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a42a57c1-44d9-4218-b188-d1fbecce4586_isr_947422_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a42a57c1-44d9-4218-b188-d1fbecce4586_isr_947422_1.stderr
parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909
parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6cdd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910
parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6df10 state=pending>
parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6eb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 38dd61d1db923aa18e71130827a76c1d
parsl.dataflow.memoization INFO: Task 912 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305
parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/894d900c-79f7-4bd7-b9a4-62392d3bf7b7_isr_947418_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/894d900c-79f7-4bd7-b9a4-62392d3bf7b7_isr_947418_54.stderr
parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912
parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: <AppFuture at 0x14a6f8d2e150 state=pending>
parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913
parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6f850 state=pending>
parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6ff50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 0afda3c81ac8620cf4847da0ac722f43
parsl.dataflow.memoization INFO: Task 915 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306
parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5913627c-b237-46d2-8e00-97694b6e7ff1_isr_947422_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5913627c-b237-46d2-8e00-97694b6e7ff1_isr_947422_57.stderr
parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915
parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6f5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916
parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7c750 state=pending>
parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7d310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 0d86a8d93a5f519cb1b6c876bd94cab1
parsl.dataflow.memoization INFO: Task 918 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307
parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/65bd3b89-01eb-4d30-be64-7835de430cfd_isr_947422_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/65bd3b89-01eb-4d30-be64-7835de430cfd_isr_947422_20.stderr
parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918
parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7d490 state=pending>
parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919
parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7e050 state=pending>
parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7e790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 821af2f544fbdf7e4520e2103ef5b8de
parsl.dataflow.memoization INFO: Task 921 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308
parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8670acad-4de0-4d03-b091-7d5048df8e5f_isr_947526_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8670acad-4de0-4d03-b091-7d5048df8e5f_isr_947526_51.stderr
parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921
parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7dd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922
parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7d9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7fb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 924 has memoization hash e98732876376d98f838a4b173a8d7183
parsl.dataflow.memoization INFO: Task 924 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309
parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce572fb3-5c92-4624-8eee-d2a9e283c548_isr_947524_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce572fb3-5c92-4624-8eee-d2a9e283c548_isr_947524_6.stderr
parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924
parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: <AppFuture at 0x14a6f8b7ff10 state=pending>
parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925
parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: <AppFuture at 0x14a6f8b84590 state=pending>
parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: <AppFuture at 0x14a6f8b84f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 96fff5f2982df0ee4f65be857aa115e3
parsl.dataflow.memoization INFO: Task 927 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310
parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/3895ab8b-10cd-4f97-8246-b3f63bb679da_isr_947523_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/3895ab8b-10cd-4f97-8246-b3f63bb679da_isr_947523_56.stderr
parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927
parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: <AppFuture at 0x14a6f8b85090 state=pending>
parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928
parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: <AppFuture at 0x14a6f8b85690 state=pending>
parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: <AppFuture at 0x14a6f8b862d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 16441f5f25af7dd7d77d08a2e2197293
parsl.dataflow.memoization INFO: Task 930 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311
parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/77192c21-e693-4289-80f1-1defc2b1224f_isr_947523_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/77192c21-e693-4289-80f1-1defc2b1224f_isr_947523_39.stderr
parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930
parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: <AppFuture at 0x14a6f8b85bd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931
parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: <AppFuture at 0x14a6f8b87150 state=pending>
parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: <AppFuture at 0x14a6f8b87710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 4571bc9a13dd825e1b26c096a5a5644b
parsl.dataflow.memoization INFO: Task 933 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312
parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b936ebc8-307c-44c6-a339-1ac91910b072_isr_947526_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b936ebc8-307c-44c6-a339-1ac91910b072_isr_947526_35.stderr
parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933
parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: <AppFuture at 0x14a6f8b87e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934
parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: <AppFuture at 0x14a6f8b94050 state=pending>
parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: <AppFuture at 0x14a6f8b94b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 936 has memoization hash f8fc1a66998367b66b0f3164534d3543
parsl.dataflow.memoization INFO: Task 936 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313
parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dfc42436-56e1-4964-8b7a-ca3ce5da2b7e_isr_947524_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dfc42436-56e1-4964-8b7a-ca3ce5da2b7e_isr_947524_26.stderr
parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936
parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: <AppFuture at 0x14a6f8b94790 state=pending>
parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937
parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: <AppFuture at 0x14a6f8b95b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: <AppFuture at 0x14a6f8b96010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 939 has memoization hash 7d92967c49995b76ce068ada78b355c4
parsl.dataflow.memoization INFO: Task 939 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314
parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/77e23278-023c-4876-9852-76cad3b6132c_isr_947524_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/77e23278-023c-4876-9852-76cad3b6132c_isr_947524_19.stderr
parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939
parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: <AppFuture at 0x14a6f8b96190 state=pending>
parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940
parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: <AppFuture at 0x14a6f8b96ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: <AppFuture at 0x14a6f8b97410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 192d44ffb5a578a9c43cdbea80d00d76
parsl.dataflow.memoization INFO: Task 942 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315
parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/85ca4416-1706-47fe-bc21-d8fc5762b6ef_isr_947528_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/85ca4416-1706-47fe-bc21-d8fc5762b6ef_isr_947528_51.stderr
parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942
parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: <AppFuture at 0x14a6f8b977d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943
parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: <AppFuture at 0x14a6f8b97890 state=pending>
parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba47d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 88ac2184b92ba20eb73b0f9f6321bbab
parsl.dataflow.memoization INFO: Task 945 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316
parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1fcf5a69-01ec-44e3-aea7-a9c647a74fe9_isr_947418_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1fcf5a69-01ec-44e3-aea7-a9c647a74fe9_isr_947418_8.stderr
parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945
parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: <AppFuture at 0x14a6f8d14e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946
parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba5490 state=pending>
parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba5bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 16a10f63f608000a832c5b5b653803b8
parsl.dataflow.memoization INFO: Task 948 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317
parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/379c4a0a-9d1a-43ca-874f-c92c9171f8b5_isr_947420_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/379c4a0a-9d1a-43ca-874f-c92c9171f8b5_isr_947420_47.stderr
parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948
parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba5910 state=pending>
parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949
parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba6ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba7050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 7968439decac3b82088885277e548216
parsl.dataflow.memoization INFO: Task 951 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318
parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/da8d0138-1ad2-46cf-9c34-f49e6a842072_isr_947523_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/da8d0138-1ad2-46cf-9c34-f49e6a842072_isr_947523_60.stderr
parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951
parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba6590 state=pending>
parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952
parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: <AppFuture at 0x14a6f8ba7f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: <AppFuture at 0x14a6f8bac510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 64f8662cc2836192739d0e71e0ad6644
parsl.dataflow.memoization INFO: Task 954 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319
parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/610da794-7c47-4d6b-b374-191cdfb2abc3_isr_947420_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/610da794-7c47-4d6b-b374-191cdfb2abc3_isr_947420_10.stderr
parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954
parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: <AppFuture at 0x14a6f8bad210 state=pending>
parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955
parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: <AppFuture at 0x14a6f8baced0 state=pending>
parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: <AppFuture at 0x14a6f8bad850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 957 has memoization hash e6705e8f76f886e6bf2edff7e8afc36b
parsl.dataflow.memoization INFO: Task 957 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320
parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/fc7e0f63-7f20-4dc1-be72-9435a1b8e8f1_isr_947524_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/fc7e0f63-7f20-4dc1-be72-9435a1b8e8f1_isr_947524_38.stderr
parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957
parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: <AppFuture at 0x14a6f8bad590 state=pending>
parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958
parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: <AppFuture at 0x14a6f8bae750 state=pending>
parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: <AppFuture at 0x14a6f8baecd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 960 has memoization hash b4d1bdc62047074676cd2254663330dc
parsl.dataflow.memoization INFO: Task 960 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321
parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6f3bbcf9-4680-4265-80a3-3c5638a82425_isr_947528_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6f3bbcf9-4680-4265-80a3-3c5638a82425_isr_947528_42.stderr
parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960
parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: <AppFuture at 0x14a6f8baf050 state=pending>
parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961
parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: <AppFuture at 0x14a6f8bafa10 state=pending>
parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: <AppFuture at 0x14a6f8bb8090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 05f376eef1dae36857a3b33b91c9687d
parsl.dataflow.memoization INFO: Task 963 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322
parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7e500f38-db63-4e56-b3b1-eba23cfa84a3_isr_947523_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7e500f38-db63-4e56-b3b1-eba23cfa84a3_isr_947523_35.stderr
parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963
parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: <AppFuture at 0x14a6f8bafed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964
parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: <AppFuture at 0x14a6f8bb88d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: <AppFuture at 0x14a6f8bb9510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 966 has memoization hash dad0c02112a0b6cb2c31b6bfc82a381b
parsl.dataflow.memoization INFO: Task 966 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323
parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4c39db58-65ad-4ce7-8527-37b007391047_isr_947524_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4c39db58-65ad-4ce7-8527-37b007391047_isr_947524_40.stderr
parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966
parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: <AppFuture at 0x14a6f8bb8e10 state=pending>
parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967
parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: <AppFuture at 0x14a6f8bb9f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: <AppFuture at 0x14a6f8bba9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 3454512b34a0a62e80b256cf59555ff6
parsl.dataflow.memoization INFO: Task 969 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324
parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/8fae28cb-2c44-432d-825b-01534f663e18_isr_947422_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/8fae28cb-2c44-432d-825b-01534f663e18_isr_947422_34.stderr
parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969
parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: <AppFuture at 0x14a6f8bba6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970
parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: <AppFuture at 0x14a6f8bbbad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: <AppFuture at 0x14a6f8bbbd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 56659ec7fc545dcfc96e2f4e66ee3a5d
parsl.dataflow.memoization INFO: Task 972 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325
parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6a2ae717-0668-4208-b3ce-ada8499b2764_isr_947523_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6a2ae717-0668-4208-b3ce-ada8499b2764_isr_947523_28.stderr
parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972
parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: <AppFuture at 0x14a6f8bbb9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973
parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: <AppFuture at 0x14a6f8bc8b90 state=pending>
parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: <AppFuture at 0x14a6f8bc9210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 975 has memoization hash f2c1910446a1d230a823071235aa0e51
parsl.dataflow.memoization INFO: Task 975 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326
parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2d637ece-fc75-41a8-b3d6-15ba08daaa7f_isr_947418_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2d637ece-fc75-41a8-b3d6-15ba08daaa7f_isr_947418_24.stderr
parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975
parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: <AppFuture at 0x14a6f8bc8f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976
parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: <AppFuture at 0x14a6f8bc9b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: <AppFuture at 0x14a6f8bca610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 978 has memoization hash 1e9812c41e53c00d8a6f0613e5cedb91
parsl.dataflow.memoization INFO: Task 978 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327
parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/14a31a42-97af-463a-95ae-243b022b261a_isr_947528_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/14a31a42-97af-463a-95ae-243b022b261a_isr_947528_50.stderr
parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978
parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: <AppFuture at 0x14a6f8bca350 state=pending>
parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979
parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: <AppFuture at 0x14a6f8bcae50 state=pending>
parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: <AppFuture at 0x14a6f8bcba10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 981 has memoization hash d7f3b069419fb5e848a7a5563d6f4364
parsl.dataflow.memoization INFO: Task 981 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328
parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3a5fa96e-70f7-4318-a76a-b45d7894df4c_isr_947420_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3a5fa96e-70f7-4318-a76a-b45d7894df4c_isr_947420_29.stderr
parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981
parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: <AppFuture at 0x14a6f8bcbfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982
parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd0650 state=pending>
parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd0e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 984 has memoization hash de20558d77329c58ba734d317d706dc0
parsl.dataflow.memoization INFO: Task 984 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329
parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d36df66-d5b3-4035-885c-650ab6531665_isr_947523_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d36df66-d5b3-4035-885c-650ab6531665_isr_947523_29.stderr
parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984
parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd0610 state=pending>
parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985
parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd1c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd21d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 4f685c5c6749c5134a9dd7d82ac30d36
parsl.dataflow.memoization INFO: Task 987 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330
parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c2126688-3c5f-40c6-8af8-1b09901651eb_isr_947420_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c2126688-3c5f-40c6-8af8-1b09901651eb_isr_947420_9.stderr
parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987
parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd2350 state=pending>
parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988
parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd2a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd3650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 95496af3bc06f3cdcd7dc49c683786b5
parsl.dataflow.memoization INFO: Task 990 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331
parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/8818464d-1c96-4760-9792-3be3dd17fd29_isr_947524_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/8818464d-1c96-4760-9792-3be3dd17fd29_isr_947524_25.stderr
parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990
parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd2ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991
parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd3b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd89d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 993 has memoization hash ed710c134bd33578e8c9cc0228ba8ef9
parsl.dataflow.memoization INFO: Task 993 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332
parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/58b45435-e135-4d0a-a5e9-45ec10a1effa_isr_947420_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/58b45435-e135-4d0a-a5e9-45ec10a1effa_isr_947420_17.stderr
parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993
parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd9310 state=pending>
parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994
parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd9290 state=pending>
parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: <AppFuture at 0x14a6f8bd9e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 996 has memoization hash cf61b9b2f69fed5f3605980995c9cd6a
parsl.dataflow.memoization INFO: Task 996 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333
parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e5c432ad-6f35-4f0e-b546-2b382d213fe3_isr_947418_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e5c432ad-6f35-4f0e-b546-2b382d213fe3_isr_947418_22.stderr
parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996
parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: <AppFuture at 0x14a6f8bda6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997
parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: <AppFuture at 0x14a6f8bda610 state=pending>
parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: <AppFuture at 0x14a6f8bdb210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 04dcf467f2b3bfe04c55838b1f9d9eae
parsl.dataflow.memoization INFO: Task 999 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334
parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/20b32f99-3ce9-489b-971a-8e43e05c33bf_isr_947422_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/20b32f99-3ce9-489b-971a-8e43e05c33bf_isr_947422_15.stderr
parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999
parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: <AppFuture at 0x14a6f8bdb510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000
parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: <AppFuture at 0x14a6f8bdbf90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: <AppFuture at 0x14a6f8be4690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 2b303f91ca2b908c664fc79981263f3f
parsl.dataflow.memoization INFO: Task 1002 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335
parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/1945028b-703d-49c6-a03d-924bc1fdaa72_isr_947524_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/1945028b-703d-49c6-a03d-924bc1fdaa72_isr_947524_3.stderr
parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002
parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5d710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003
parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: <AppFuture at 0x14a6f8be5310 state=pending>
parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: <AppFuture at 0x14a6f8be5b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 0ec06190d5e344ca1756cee7bcbaa433
parsl.dataflow.memoization INFO: Task 1005 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336
parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/08ccaa4d-b0e2-4efe-ad6e-76e08d5040d0_isr_947528_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/08ccaa4d-b0e2-4efe-ad6e-76e08d5040d0_isr_947528_30.stderr
parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005
parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: <AppFuture at 0x14a6f8be5790 state=pending>
parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006
parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: <AppFuture at 0x14a6f8be63d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: <AppFuture at 0x14a6f8be7050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 4ad920edea67f29587b4bc7f6161f941
parsl.dataflow.memoization INFO: Task 1008 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337
parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/36aceef4-2f39-43b2-b1cd-9f8d5efd5b28_isr_947420_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/36aceef4-2f39-43b2-b1cd-9f8d5efd5b28_isr_947420_6.stderr
parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008
parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: <AppFuture at 0x14a6f8be7190 state=pending>
parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009
parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: <AppFuture at 0x14a6f8be7650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf04d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 9970b4da2a25ddd174f4d033b6cbe0a9
parsl.dataflow.memoization INFO: Task 1011 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338
parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11237c37-a4a9-45b7-920f-a7ee8e806b0e_isr_947528_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11237c37-a4a9-45b7-920f-a7ee8e806b0e_isr_947528_38.stderr
parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011
parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf0610 state=pending>
parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012
parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf12d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf1790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash d29198dc166182690e6e5a5fe34dd31a
parsl.dataflow.memoization INFO: Task 1014 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339
parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2ae1a682-b0dd-450c-86a0-7cd7ed3b7067_isr_947420_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2ae1a682-b0dd-450c-86a0-7cd7ed3b7067_isr_947420_40.stderr
parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014
parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf1d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015
parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf2850 state=pending>
parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf2c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash c2ca3da327baafa18553cb7db54c6e83
parsl.dataflow.memoization INFO: Task 1017 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340
parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2c13f486-4e02-4279-a26a-e6ae0d51e50a_isr_947420_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2c13f486-4e02-4279-a26a-e6ae0d51e50a_isr_947420_5.stderr
parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017
parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf3490 state=pending>
parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018
parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf3450 state=pending>
parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf3d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 6523f3205a3e35edebc2ecfa37e86dda
parsl.dataflow.memoization INFO: Task 1020 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341
parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/02cbf624-9284-45f6-b962-a3d80f4b892f_isr_947418_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/02cbf624-9284-45f6-b962-a3d80f4b892f_isr_947418_46.stderr
parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020
parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: <AppFuture at 0x14a6f8bf33d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021
parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: <AppFuture at 0x14a6f8c00490 state=pending>
parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: <AppFuture at 0x14a6f8c01450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash a3cfc32a46a9f33ae54ff70a14cbfd6c
parsl.dataflow.memoization INFO: Task 1023 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342
parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cc9061b-b6ed-45ab-b5f9-6b974ab94b9a_isr_947418_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cc9061b-b6ed-45ab-b5f9-6b974ab94b9a_isr_947418_20.stderr
parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023
parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: <AppFuture at 0x14a6f8c015d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024
parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: <AppFuture at 0x14a6f8c022d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: <AppFuture at 0x14a6f8c02850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash b83c0ac100ab4d6b217f2083e1492f88
parsl.dataflow.memoization INFO: Task 1026 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343
parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/39058b2f-4771-403c-93a1-825e28c63716_isr_947418_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/39058b2f-4771-403c-93a1-825e28c63716_isr_947418_11.stderr
parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026
parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: <AppFuture at 0x14a6f8c01c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027
parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: <AppFuture at 0x14a6f8c03710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: <AppFuture at 0x14a6f8c03c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash db3d3a51d05189e60588beca8e9ab550
parsl.dataflow.memoization INFO: Task 1029 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344
parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/027cd8bb-5bc9-4011-b470-679a6c175884_isr_947528_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/027cd8bb-5bc9-4011-b470-679a6c175884_isr_947528_13.stderr
parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029
parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: <AppFuture at 0x14a6f8c03f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030
parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: <AppFuture at 0x14a6f8c04b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: <AppFuture at 0x14a6f8c05050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash f63b00e0c0bb4c14b986623afdb180c2
parsl.dataflow.memoization INFO: Task 1032 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345
parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2c877851-b7fc-4f0f-9128-c84d057240a8_isr_947422_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2c877851-b7fc-4f0f-9128-c84d057240a8_isr_947422_50.stderr
parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032
parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: <AppFuture at 0x14a6f8c04410 state=pending>
parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033
parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: <AppFuture at 0x14a6f8c05810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: <AppFuture at 0x14a6f8c06450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 8d089667f6f744ce83d6f5d74d5fc890
parsl.dataflow.memoization INFO: Task 1035 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346
parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/55220a37-7c5d-4d58-b1b6-2aaa3c43ee5d_isr_947524_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/55220a37-7c5d-4d58-b1b6-2aaa3c43ee5d_isr_947524_10.stderr
parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035
parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: <AppFuture at 0x14a6f8c05910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036
parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: <AppFuture at 0x14a6f8c06e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: <AppFuture at 0x14a6f8c07810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 0a5fad219b1b570e944d73ade232a210
parsl.dataflow.memoization INFO: Task 1038 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347
parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0746290a-d097-4b80-b294-e6ad12187d37_isr_947526_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0746290a-d097-4b80-b294-e6ad12187d37_isr_947526_41.stderr
parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038
parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: <AppFuture at 0x14a6f8c07c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039
parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: <AppFuture at 0x14a6f8c10650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: <AppFuture at 0x14a6f8c10c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 867ab45e5e196fbd8bf7336a3f5a8ff7
parsl.dataflow.memoization INFO: Task 1041 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348
parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bcfb6b0b-cd6d-4e58-b399-7266b633e673_isr_947526_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bcfb6b0b-cd6d-4e58-b399-7266b633e673_isr_947526_17.stderr
parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041
parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: <AppFuture at 0x14a6f8c109d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042
parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: <AppFuture at 0x14a6f8c11890 state=pending>
parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: <AppFuture at 0x14a6f8c120d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 4d781dd616df96e1878b42205eab3138
parsl.dataflow.memoization INFO: Task 1044 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349
parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/fa97f8bd-b59e-4781-8120-1643d6611d51_isr_947422_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/fa97f8bd-b59e-4781-8120-1643d6611d51_isr_947422_6.stderr
parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044
parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: <AppFuture at 0x14a6f8c114d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045
parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: <AppFuture at 0x14a6f8c12910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: <AppFuture at 0x14a6f8c13510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 5c9e29fc65659433d84f06297178505e
parsl.dataflow.memoization INFO: Task 1047 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350
parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/01faa5ef-1df7-4fa7-a0e9-fa16e78a7c63_isr_947420_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/01faa5ef-1df7-4fa7-a0e9-fa16e78a7c63_isr_947420_57.stderr
parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047
parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: <AppFuture at 0x14a6f8c13250 state=pending>
parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048
parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: <AppFuture at 0x14a6f8c13f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1c8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash 2a2e26729177f4d370e4b242cb8477fa
parsl.dataflow.memoization INFO: Task 1050 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351
parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9b36a470-55a3-44bc-952c-fa64970b73a4_isr_947528_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9b36a470-55a3-44bc-952c-fa64970b73a4_isr_947528_49.stderr
parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050
parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: <AppFuture at 0x14a6f8d5d1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051
parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1d590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1dcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash d90397a681a3dfb0a2ab6a82b9d0c659
parsl.dataflow.memoization INFO: Task 1053 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352
parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5518c18a-e637-4d3c-b468-b01a43faafcb_isr_947523_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5518c18a-e637-4d3c-b468-b01a43faafcb_isr_947523_7.stderr
parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053
parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1d950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054
parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1ed10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1f250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 6017e5db79d732eddd5b3c3e7411e002
parsl.dataflow.memoization INFO: Task 1056 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353
parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bf077fa1-f099-46e9-b5a1-24f6ecf8e268_isr_947528_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bf077fa1-f099-46e9-b5a1-24f6ecf8e268_isr_947528_23.stderr
parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056
parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1f3d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057
parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1fe10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: <AppFuture at 0x14a6f8c285d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 00451eaa80cf740c8baa0972b965a70e
parsl.dataflow.memoization INFO: Task 1059 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354
parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6f824794-fa1c-41ec-a186-2ce2265aea78_isr_947526_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6f824794-fa1c-41ec-a186-2ce2265aea78_isr_947526_39.stderr
parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059
parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: <AppFuture at 0x14a6f8c1e290 state=pending>
parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060
parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: <AppFuture at 0x14a6f8c291d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: <AppFuture at 0x14a6f8c29950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 8f6508dc80de3c1c967e84cc6d044d04
parsl.dataflow.memoization INFO: Task 1062 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355
parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/d9da1282-7531-4bd6-bc5d-496888e1bb23_isr_947528_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/d9da1282-7531-4bd6-bc5d-496888e1bb23_isr_947528_3.stderr
parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062
parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: <AppFuture at 0x14a6f8c29cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063
parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: <AppFuture at 0x14a6f8c2a790 state=pending>
parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: <AppFuture at 0x14a6f8c2ae50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 9df982db57fd977e40c131225c45241c
parsl.dataflow.memoization INFO: Task 1065 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356
parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ff61384-0bbd-4c56-b95f-6b89c97a7747_isr_947524_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ff61384-0bbd-4c56-b95f-6b89c97a7747_isr_947524_5.stderr
parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065
parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: <AppFuture at 0x14a6f8c28d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066
parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: <AppFuture at 0x14a6f8c2b5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: <AppFuture at 0x14a6f8c38110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash c4bce17763315288a6fec69ca9acb745
parsl.dataflow.memoization INFO: Task 1068 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357
parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/332fe5e0-6d63-4c4d-a34a-c43f05991797_isr_947420_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/332fe5e0-6d63-4c4d-a34a-c43f05991797_isr_947420_49.stderr
parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068
parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: <AppFuture at 0x14a6f8c2bb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069
parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: <AppFuture at 0x14a6f8c38a50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: <AppFuture at 0x14a6f8c39590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash 5f7a6be46a8efc2054091f6efb353b35
parsl.dataflow.memoization INFO: Task 1071 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358
parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/44ac5511-67eb-4236-aac8-3431a8930868_isr_947524_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/44ac5511-67eb-4236-aac8-3431a8930868_isr_947524_22.stderr
parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071
parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: <AppFuture at 0x14a6f8c39710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072
parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: <AppFuture at 0x14a6f8c39e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: <AppFuture at 0x14a6f8c3aa10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 3948c4410ec363f3ce182dce4b69162c
parsl.dataflow.memoization INFO: Task 1074 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359
parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a5e1d854-49ff-4ea8-a3c3-980d435604df_isr_947422_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a5e1d854-49ff-4ea8-a3c3-980d435604df_isr_947422_11.stderr
parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074
parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: <AppFuture at 0x14a6f8c3ab90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075
parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: <AppFuture at 0x14a6f8c3b250 state=pending>
parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: <AppFuture at 0x14a6f8c3bdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash e1dca6cb617849199991e375dcddde0d
parsl.dataflow.memoization INFO: Task 1077 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360
parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/877b7d36-95c9-4d12-9221-1cf7875591c2_isr_947528_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/877b7d36-95c9-4d12-9221-1cf7875591c2_isr_947528_12.stderr
parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077
parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: <AppFuture at 0x14a6f8c3ba90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078
parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: <AppFuture at 0x14a6f8c44650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: <AppFuture at 0x14a6f8c45190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash f0d8bab40f2708ed582a8ebb95aa504f
parsl.dataflow.memoization INFO: Task 1080 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361
parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2ec66d38-7a23-4fe7-b20b-55c54ad09b3b_isr_947524_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2ec66d38-7a23-4fe7-b20b-55c54ad09b3b_isr_947524_47.stderr
parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080
parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: <AppFuture at 0x14a6f8c45310 state=pending>
parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081
parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: <AppFuture at 0x14a6f8c45e50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: <AppFuture at 0x14a6f8c46690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 073667e9cb134ca0a144aeec912cdaf1
parsl.dataflow.memoization INFO: Task 1083 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362
parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/ff65a48c-8765-46a4-a19d-fcca10e7e555_isr_947528_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/ff65a48c-8765-46a4-a19d-fcca10e7e555_isr_947528_24.stderr
parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083
parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: <AppFuture at 0x14a6f8c45910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084
parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: <AppFuture at 0x14a6f8c47550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: <AppFuture at 0x14a6f8c47ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 22bf10b362a2ae595208c9f459ac83ca
parsl.dataflow.memoization INFO: Task 1086 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363
parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/511843d9-ed88-4821-94cf-8ec8ae0219ba_isr_947528_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/511843d9-ed88-4821-94cf-8ec8ae0219ba_isr_947528_9.stderr
parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086
parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: <AppFuture at 0x14a6f8c47e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087
parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4c9d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4ce10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 0217a49fcb7143a36ab62b2b4ae147d8
parsl.dataflow.memoization INFO: Task 1089 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364
parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/85cb5bb9-7b86-4098-9f58-f6a151a2d5f5_isr_947524_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/85cb5bb9-7b86-4098-9f58-f6a151a2d5f5_isr_947524_56.stderr
parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089
parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4d7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090
parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4d750 state=pending>
parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4e1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 18f9b2e614f019f6e2f3440f253f85d4
parsl.dataflow.memoization INFO: Task 1092 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365
parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d18447df-c3a7-49f5-a2ce-76ed08306afd_isr_947420_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d18447df-c3a7-49f5-a2ce-76ed08306afd_isr_947420_54.stderr
parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092
parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4e550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093
parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4f1d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4f690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash e896ce2a2c102a93a842f862736e24d7
parsl.dataflow.memoization INFO: Task 1095 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366
parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/053f1a53-fbc6-4f05-9978-050ea50222ba_isr_947524_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/053f1a53-fbc6-4f05-9978-050ea50222ba_isr_947524_12.stderr
parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095
parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4f7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096
parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: <AppFuture at 0x14a6f8c4fb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: <AppFuture at 0x14a6f8c54a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash 5d3aa1bcf1fbf61f5e6ebadb14ea1ad3
parsl.dataflow.memoization INFO: Task 1098 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367
parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d32aa41-421d-4d88-a06c-7f3e70e6ee8c_isr_947523_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d32aa41-421d-4d88-a06c-7f3e70e6ee8c_isr_947523_37.stderr
parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098
parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: <AppFuture at 0x14a6f8c45ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099
parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: <AppFuture at 0x14a6f8c552d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: <AppFuture at 0x14a6f8c55e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash e9aa001273d41bd6ea3b3d4bbdd2622c
parsl.dataflow.memoization INFO: Task 1101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368
parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/be4c5dab-2e27-4562-9cbf-62c8c0422fc7_isr_947422_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/be4c5dab-2e27-4562-9cbf-62c8c0422fc7_isr_947422_3.stderr
parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101
parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: <AppFuture at 0x14a6f8c55250 state=pending>
parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102
parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: <AppFuture at 0x14a6f8c56c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: <AppFuture at 0x14a6f8c57290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 0d0e1b9aab5e7124dc61202f682c9123
parsl.dataflow.memoization INFO: Task 1104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369
parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7984abcb-29dd-4f03-a0bd-db439191a2d8_isr_947422_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7984abcb-29dd-4f03-a0bd-db439191a2d8_isr_947422_43.stderr
parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104
parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: <AppFuture at 0x14a6f8c57610 state=pending>
parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105
parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: <AppFuture at 0x14a6f8c57d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: <AppFuture at 0x14a6f8c64750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash c3bec5f39013924f6a846299bf1cbfb1
parsl.dataflow.memoization INFO: Task 1107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370
parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d1cc225f-d2cc-427d-92b4-2af9c478e713_isr_947418_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d1cc225f-d2cc-427d-92b4-2af9c478e713_isr_947418_40.stderr
parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107
parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: <AppFuture at 0x14a6f8c57750 state=pending>
parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108
parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: <AppFuture at 0x14a6f8c653d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: <AppFuture at 0x14a6f8c65b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 1457a09fd031adc1ba940024cf2b0d79
parsl.dataflow.memoization INFO: Task 1110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371
parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/6b904d7d-73e4-4e61-8bd6-724d3de36818_isr_947418_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/6b904d7d-73e4-4e61-8bd6-724d3de36818_isr_947418_50.stderr
parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110
parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: <AppFuture at 0x14a6f8c65c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111
parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: <AppFuture at 0x14a6f8c663d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: <AppFuture at 0x14a6f8c66fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 7470ce5e6edf11c42cc7838cbf7ed10b
parsl.dataflow.memoization INFO: Task 1113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372
parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/647396c0-fdf5-4fea-b643-8c679fb15bd8_isr_947526_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/647396c0-fdf5-4fea-b643-8c679fb15bd8_isr_947526_7.stderr
parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113
parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: <AppFuture at 0x14a6f8c67150 state=pending>
parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114
parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: <AppFuture at 0x14a6f8c677d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1116 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1116 set to pending state with AppFuture: <AppFuture at 0x14a6f8c70450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1116 has memoization hash 6206e8002191f38a65109d15e4dcc3b2
parsl.dataflow.memoization INFO: Task 1116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1116 try 0 launched on executor multi with executor id 373
parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5036acce-23b5-4cbd-8ecb-cc1fb1239550_isr_947422_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5036acce-23b5-4cbd-8ecb-cc1fb1239550_isr_947422_9.stderr
parsl.dataflow.dflow DEBUG: Task 1117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1117 submitted for App characterizeImage, waiting on task 1116
parsl.dataflow.dflow DEBUG: Task 1117 set to pending state with AppFuture: <AppFuture at 0x14a6f8c705d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1117 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1118 submitted for App calibrate, waiting on task 1117
parsl.dataflow.dflow DEBUG: Task 1118 set to pending state with AppFuture: <AppFuture at 0x14a6f8c70150 state=pending>
parsl.dataflow.dflow DEBUG: Task 1118 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1119 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1119 set to pending state with AppFuture: <AppFuture at 0x14a6f8c71790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1119 has memoization hash 283e5863a922796219400c4c0f8f913a
parsl.dataflow.memoization INFO: Task 1119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1119 try 0 launched on executor multi with executor id 374
parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5a92f97f-7806-4fd3-948b-39a10df55cdd_isr_947526_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5a92f97f-7806-4fd3-948b-39a10df55cdd_isr_947526_45.stderr
parsl.dataflow.dflow DEBUG: Task 1120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1120 submitted for App characterizeImage, waiting on task 1119
parsl.dataflow.dflow DEBUG: Task 1120 set to pending state with AppFuture: <AppFuture at 0x14a6f8c70c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1120 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1121 submitted for App calibrate, waiting on task 1120
parsl.dataflow.dflow DEBUG: Task 1121 set to pending state with AppFuture: <AppFuture at 0x14a6f8c71f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1121 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1122 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1122 set to pending state with AppFuture: <AppFuture at 0x14a6f8c72bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1122 has memoization hash b0b09656de8fab465b2e30f40248979f
parsl.dataflow.memoization INFO: Task 1122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1122 try 0 launched on executor multi with executor id 375
parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/492987d5-0a90-468a-b551-ccadbf082746_isr_947418_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/492987d5-0a90-468a-b551-ccadbf082746_isr_947418_23.stderr
parsl.dataflow.dflow DEBUG: Task 1123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1123 submitted for App characterizeImage, waiting on task 1122
parsl.dataflow.dflow DEBUG: Task 1123 set to pending state with AppFuture: <AppFuture at 0x14a6f8c72910 state=pending>
parsl.dataflow.dflow DEBUG: Task 1123 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1124 submitted for App calibrate, waiting on task 1123
parsl.dataflow.dflow DEBUG: Task 1124 set to pending state with AppFuture: <AppFuture at 0x14a6f8c73650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1124 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1125 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1125 set to pending state with AppFuture: <AppFuture at 0x14a6f8c73f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1125 has memoization hash fba605b43113823a416d856cf4afccc1
parsl.dataflow.memoization INFO: Task 1125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1125 try 0 launched on executor multi with executor id 376
parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0edd9aad-6198-40f7-b475-aef4cd3e33f1_isr_947422_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0edd9aad-6198-40f7-b475-aef4cd3e33f1_isr_947422_25.stderr
parsl.dataflow.dflow DEBUG: Task 1126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1126 submitted for App characterizeImage, waiting on task 1125
parsl.dataflow.dflow DEBUG: Task 1126 set to pending state with AppFuture: <AppFuture at 0x14a6f8c73cd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1126 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1127 submitted for App calibrate, waiting on task 1126
parsl.dataflow.dflow DEBUG: Task 1127 set to pending state with AppFuture: <AppFuture at 0x14a6f8a80650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1127 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1128 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1128 set to pending state with AppFuture: <AppFuture at 0x14a6f8a813d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1128 has memoization hash 1f72bbf0801b540aa7206b5a986250db
parsl.dataflow.memoization INFO: Task 1128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1128 try 0 launched on executor multi with executor id 377
parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/187dd995-7b5d-4c95-8f47-553ce9c9171c_isr_947526_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/187dd995-7b5d-4c95-8f47-553ce9c9171c_isr_947526_58.stderr
parsl.dataflow.dflow DEBUG: Task 1129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1129 submitted for App characterizeImage, waiting on task 1128
parsl.dataflow.dflow DEBUG: Task 1129 set to pending state with AppFuture: <AppFuture at 0x14a6f8a807d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1129 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1130 submitted for App calibrate, waiting on task 1129
parsl.dataflow.dflow DEBUG: Task 1130 set to pending state with AppFuture: <AppFuture at 0x14a6f8a822d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1130 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1131 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1131 set to pending state with AppFuture: <AppFuture at 0x14a6f8a82850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1131 has memoization hash 046fe30d94b2a03f72d83d0a4ff9945c
parsl.dataflow.memoization INFO: Task 1131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1131 try 0 launched on executor multi with executor id 378
parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a0c9aa73-8373-4555-8d38-a8838abdb12a_isr_947418_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a0c9aa73-8373-4555-8d38-a8838abdb12a_isr_947418_6.stderr
parsl.dataflow.dflow DEBUG: Task 1132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1132 submitted for App characterizeImage, waiting on task 1131
parsl.dataflow.dflow DEBUG: Task 1132 set to pending state with AppFuture: <AppFuture at 0x14a6f8a81d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1132 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1133 submitted for App calibrate, waiting on task 1132
parsl.dataflow.dflow DEBUG: Task 1133 set to pending state with AppFuture: <AppFuture at 0x14a6f8a830d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1133 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1134 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1134 set to pending state with AppFuture: <AppFuture at 0x14a6f8a83c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1134 has memoization hash c3179a7e48a74379245b6674b1d5566c
parsl.dataflow.memoization INFO: Task 1134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1134 try 0 launched on executor multi with executor id 379
parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/34ea5f45-5718-493e-b824-9158392eb203_isr_947422_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/34ea5f45-5718-493e-b824-9158392eb203_isr_947422_18.stderr
parsl.dataflow.dflow DEBUG: Task 1135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1135 submitted for App characterizeImage, waiting on task 1134
parsl.dataflow.dflow DEBUG: Task 1135 set to pending state with AppFuture: <AppFuture at 0x14a6f8a82c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1135 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1136 submitted for App calibrate, waiting on task 1135
parsl.dataflow.dflow DEBUG: Task 1136 set to pending state with AppFuture: <AppFuture at 0x14a6f8a847d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1136 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1137 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1137 set to pending state with AppFuture: <AppFuture at 0x14a6f8a85010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1137 has memoization hash abf2d9913bed6fb34dcb3e33a5adeb82
parsl.dataflow.memoization INFO: Task 1137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1137 try 0 launched on executor multi with executor id 380
parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5015b7e4-b22e-4e79-8f5c-c2708d27482c_isr_947418_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5015b7e4-b22e-4e79-8f5c-c2708d27482c_isr_947418_36.stderr
parsl.dataflow.dflow DEBUG: Task 1138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1138 submitted for App characterizeImage, waiting on task 1137
parsl.dataflow.dflow DEBUG: Task 1138 set to pending state with AppFuture: <AppFuture at 0x14a6f8a84610 state=pending>
parsl.dataflow.dflow DEBUG: Task 1138 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1139 submitted for App calibrate, waiting on task 1138
parsl.dataflow.dflow DEBUG: Task 1139 set to pending state with AppFuture: <AppFuture at 0x14a6f8a85810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1139 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1140 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1140 set to pending state with AppFuture: <AppFuture at 0x14a6f8a863d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1140 has memoization hash 664d827242976580cf4919434540b14f
parsl.dataflow.memoization INFO: Task 1140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1140 try 0 launched on executor multi with executor id 381
parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b71d7abb-1230-4b2d-8ff4-b09c86850285_isr_947528_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b71d7abb-1230-4b2d-8ff4-b09c86850285_isr_947528_55.stderr
parsl.dataflow.dflow DEBUG: Task 1141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1141 submitted for App characterizeImage, waiting on task 1140
parsl.dataflow.dflow DEBUG: Task 1141 set to pending state with AppFuture: <AppFuture at 0x14a6f8a858d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1141 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1142 submitted for App calibrate, waiting on task 1141
parsl.dataflow.dflow DEBUG: Task 1142 set to pending state with AppFuture: <AppFuture at 0x14a6f8a87350 state=pending>
parsl.dataflow.dflow DEBUG: Task 1142 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1143 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1143 set to pending state with AppFuture: <AppFuture at 0x14a6f8a878d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1143 has memoization hash b98ba387f7fbb9ea9f49de77c19fe1d3
parsl.dataflow.memoization INFO: Task 1143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1143 try 0 launched on executor multi with executor id 382
parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/05b1e838-c5f6-4485-90a6-39ac4a009807_isr_947528_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/05b1e838-c5f6-4485-90a6-39ac4a009807_isr_947528_17.stderr
parsl.dataflow.dflow DEBUG: Task 1144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1144 submitted for App characterizeImage, waiting on task 1143
parsl.dataflow.dflow DEBUG: Task 1144 set to pending state with AppFuture: <AppFuture at 0x14a6f8a87dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1144 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1145 submitted for App calibrate, waiting on task 1144
parsl.dataflow.dflow DEBUG: Task 1145 set to pending state with AppFuture: <AppFuture at 0x14a6f8a94710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1145 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1146 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1146 set to pending state with AppFuture: <AppFuture at 0x14a6f8a94c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1146 has memoization hash 16655c8047183f0e4d5cff275612a2ec
parsl.dataflow.memoization INFO: Task 1146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1146 try 0 launched on executor multi with executor id 383
parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0cc3d9ce-d212-4df0-ba37-c5c4e16b0cc2_isr_947524_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0cc3d9ce-d212-4df0-ba37-c5c4e16b0cc2_isr_947524_9.stderr
parsl.dataflow.dflow DEBUG: Task 1147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1147 submitted for App characterizeImage, waiting on task 1146
parsl.dataflow.dflow DEBUG: Task 1147 set to pending state with AppFuture: <AppFuture at 0x14a6f8a94dd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1147 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1148 submitted for App calibrate, waiting on task 1147
parsl.dataflow.dflow DEBUG: Task 1148 set to pending state with AppFuture: <AppFuture at 0x14a6f8a95c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1148 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1149 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1149 set to pending state with AppFuture: <AppFuture at 0x14a6f8a96050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1149 has memoization hash 67bbcf49c6e91d921438f3063667ca4e
parsl.dataflow.memoization INFO: Task 1149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1149 try 0 launched on executor multi with executor id 384
parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/57aaf85a-ce82-4106-976c-0df6bb035413_isr_947526_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/57aaf85a-ce82-4106-976c-0df6bb035413_isr_947526_48.stderr
parsl.dataflow.dflow DEBUG: Task 1150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1150 submitted for App characterizeImage, waiting on task 1149
parsl.dataflow.dflow DEBUG: Task 1150 set to pending state with AppFuture: <AppFuture at 0x14a6f8a95490 state=pending>
parsl.dataflow.dflow DEBUG: Task 1150 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1151 submitted for App calibrate, waiting on task 1150
parsl.dataflow.dflow DEBUG: Task 1151 set to pending state with AppFuture: <AppFuture at 0x14a6f8a97090 state=pending>
parsl.dataflow.dflow DEBUG: Task 1151 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1152 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1152 set to pending state with AppFuture: <AppFuture at 0x14a6f8a974d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1152 has memoization hash f6a549fe000824c764c6fe919b6ef92e
parsl.dataflow.memoization INFO: Task 1152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1152 try 0 launched on executor multi with executor id 385
parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/e5d69ca6-5678-4015-bf79-0e2abbae3b9c_isr_947528_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/e5d69ca6-5678-4015-bf79-0e2abbae3b9c_isr_947528_39.stderr
parsl.dataflow.dflow DEBUG: Task 1153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1153 submitted for App characterizeImage, waiting on task 1152
parsl.dataflow.dflow DEBUG: Task 1153 set to pending state with AppFuture: <AppFuture at 0x14a6f8a97210 state=pending>
parsl.dataflow.dflow DEBUG: Task 1153 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1154 submitted for App calibrate, waiting on task 1153
parsl.dataflow.dflow DEBUG: Task 1154 set to pending state with AppFuture: <AppFuture at 0x14a6f8a979d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1154 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1155 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1155 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa08d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1155 has memoization hash b103d04e3678dab0ce87d1431d240936
parsl.dataflow.memoization INFO: Task 1155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1155 try 0 launched on executor multi with executor id 386
parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/cf778f2d-4d8d-49ec-8ff1-f4ae7b31ccf4_isr_947526_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/cf778f2d-4d8d-49ec-8ff1-f4ae7b31ccf4_isr_947526_36.stderr
parsl.dataflow.dflow DEBUG: Task 1156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1156 submitted for App characterizeImage, waiting on task 1155
parsl.dataflow.dflow DEBUG: Task 1156 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa0610 state=pending>
parsl.dataflow.dflow DEBUG: Task 1156 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1157 submitted for App calibrate, waiting on task 1156
parsl.dataflow.dflow DEBUG: Task 1157 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa0f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1157 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1158 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1158 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa1c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1158 has memoization hash eee489a78d619cbf4a5203590f11f474
parsl.dataflow.memoization INFO: Task 1158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1158 try 0 launched on executor multi with executor id 387
parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1e8ca4da-8f92-49fe-b267-157da6a1ac90_isr_947528_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1e8ca4da-8f92-49fe-b267-157da6a1ac90_isr_947528_29.stderr
parsl.dataflow.dflow DEBUG: Task 1159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1159 submitted for App characterizeImage, waiting on task 1158
parsl.dataflow.dflow DEBUG: Task 1159 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa2c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1159 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1160 submitted for App calibrate, waiting on task 1159
parsl.dataflow.dflow DEBUG: Task 1160 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa2b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1160 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1161 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1161 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa3110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1161 has memoization hash e38c343c0d4f090bbcb94680144ed72e
parsl.dataflow.memoization INFO: Task 1161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1161 try 0 launched on executor multi with executor id 388
parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f9deedfb-cc13-402b-8d48-2aba05dcdaf6_isr_947526_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f9deedfb-cc13-402b-8d48-2aba05dcdaf6_isr_947526_18.stderr
parsl.dataflow.dflow DEBUG: Task 1162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1162 submitted for App characterizeImage, waiting on task 1161
parsl.dataflow.dflow DEBUG: Task 1162 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa2510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1162 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1163 submitted for App calibrate, waiting on task 1162
parsl.dataflow.dflow DEBUG: Task 1163 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa3d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1163 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1164 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1164 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab0590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1164 has memoization hash c30efefeba6ae676f7bd0b13b384ab58
parsl.dataflow.memoization INFO: Task 1164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1164 try 0 launched on executor multi with executor id 389
parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9273f92f-3917-47e8-853d-d14254b63d2c_isr_947526_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9273f92f-3917-47e8-853d-d14254b63d2c_isr_947526_23.stderr
parsl.dataflow.dflow DEBUG: Task 1165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1165 submitted for App characterizeImage, waiting on task 1164
parsl.dataflow.dflow DEBUG: Task 1165 set to pending state with AppFuture: <AppFuture at 0x14a6f8aa02d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1165 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1166 submitted for App calibrate, waiting on task 1165
parsl.dataflow.dflow DEBUG: Task 1166 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab01d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1166 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1167 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1167 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab1950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1167 has memoization hash ad8395a20769f10258329eb9d07f5e03
parsl.dataflow.memoization INFO: Task 1167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1167 try 0 launched on executor multi with executor id 390
parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/69f18057-7f5c-4163-9b26-9409c3233160_isr_947528_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/69f18057-7f5c-4163-9b26-9409c3233160_isr_947528_35.stderr
parsl.dataflow.dflow DEBUG: Task 1168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1168 submitted for App characterizeImage, waiting on task 1167
parsl.dataflow.dflow DEBUG: Task 1168 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab1ad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1168 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1169 submitted for App calibrate, waiting on task 1168
parsl.dataflow.dflow DEBUG: Task 1169 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab2650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1169 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1170 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1170 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab2dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1170 has memoization hash 60b08279625e05391926f02474447c4c
parsl.dataflow.memoization INFO: Task 1170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1170 try 0 launched on executor multi with executor id 391
parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5bba55d6-93c9-480f-a742-a5044daabc08_isr_947526_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5bba55d6-93c9-480f-a742-a5044daabc08_isr_947526_43.stderr
parsl.dataflow.dflow DEBUG: Task 1171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1171 submitted for App characterizeImage, waiting on task 1170
parsl.dataflow.dflow DEBUG: Task 1171 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab2f50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1171 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1172 submitted for App calibrate, waiting on task 1171
parsl.dataflow.dflow DEBUG: Task 1172 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab3c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1172 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1173 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1173 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab4050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1173 has memoization hash 66586965de63363794a8c706b82ea42e
parsl.dataflow.memoization INFO: Task 1173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1173 try 0 launched on executor multi with executor id 392
parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/17d365dc-aa36-472b-b44b-fea993fea34e_isr_947528_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/17d365dc-aa36-472b-b44b-fea993fea34e_isr_947528_28.stderr
parsl.dataflow.dflow DEBUG: Task 1174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1174 submitted for App characterizeImage, waiting on task 1173
parsl.dataflow.dflow DEBUG: Task 1174 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab3ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1174 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1175 submitted for App calibrate, waiting on task 1174
parsl.dataflow.dflow DEBUG: Task 1175 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab4b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1175 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1176 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1176 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab5690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1176 has memoization hash 2975998ba256bbb900ae5854cfea7f91
parsl.dataflow.memoization INFO: Task 1176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1176 try 0 launched on executor multi with executor id 393
parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0647087a-77a8-426a-821c-ebc3f82c495e_isr_947526_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0647087a-77a8-426a-821c-ebc3f82c495e_isr_947526_34.stderr
parsl.dataflow.dflow DEBUG: Task 1177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1177 submitted for App characterizeImage, waiting on task 1176
parsl.dataflow.dflow DEBUG: Task 1177 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab6110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1177 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1178 submitted for App calibrate, waiting on task 1177
parsl.dataflow.dflow DEBUG: Task 1178 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab5e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1178 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1179 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1179 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab6b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1179 has memoization hash 3b2f4c6cae672b6b4088275f8532c43a
parsl.dataflow.memoization INFO: Task 1179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1179 try 0 launched on executor multi with executor id 394
parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fb97dcfe-0a08-4bea-848b-501039319d2c_isr_947523_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fb97dcfe-0a08-4bea-848b-501039319d2c_isr_947523_12.stderr
parsl.dataflow.dflow DEBUG: Task 1180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1180 submitted for App characterizeImage, waiting on task 1179
parsl.dataflow.dflow DEBUG: Task 1180 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab6c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1180 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1181 submitted for App calibrate, waiting on task 1180
parsl.dataflow.dflow DEBUG: Task 1181 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab77d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1181 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1182 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1182 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab7fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1182 has memoization hash 129f7b4be228a1e3a3aee86c058dc3d8
parsl.dataflow.memoization INFO: Task 1182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1182 try 0 launched on executor multi with executor id 395
parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/27bff7e7-d582-479a-9696-25c923917ff1_isr_947528_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/27bff7e7-d582-479a-9696-25c923917ff1_isr_947528_15.stderr
parsl.dataflow.dflow DEBUG: Task 1183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1183 submitted for App characterizeImage, waiting on task 1182
parsl.dataflow.dflow DEBUG: Task 1183 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab7c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1183 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1184 submitted for App calibrate, waiting on task 1183
parsl.dataflow.dflow DEBUG: Task 1184 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac0a10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1184 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1185 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1185 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac12d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1185 has memoization hash bed0e4d0236aafa0ce751ffe4c86aca6
parsl.dataflow.memoization INFO: Task 1185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1185 try 0 launched on executor multi with executor id 396
parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/22888194-e37e-4101-a08b-785cd98494d7_isr_947524_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/22888194-e37e-4101-a08b-785cd98494d7_isr_947524_35.stderr
parsl.dataflow.dflow DEBUG: Task 1186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1186 submitted for App characterizeImage, waiting on task 1185
parsl.dataflow.dflow DEBUG: Task 1186 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac0710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1186 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1187 submitted for App calibrate, waiting on task 1186
parsl.dataflow.dflow DEBUG: Task 1187 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac1f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1187 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1188 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1188 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac2750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1188 has memoization hash e894053330332ca89867629e7e54ed3e
parsl.dataflow.memoization INFO: Task 1188 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1188 try 0 launched on executor multi with executor id 397
parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/3b9dd0a8-3078-4f02-a4b7-0e936e2e04cb_isr_947422_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/3b9dd0a8-3078-4f02-a4b7-0e936e2e04cb_isr_947422_7.stderr
parsl.dataflow.dflow DEBUG: Task 1189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1189 submitted for App characterizeImage, waiting on task 1188
parsl.dataflow.dflow DEBUG: Task 1189 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac23d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1189 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1190 submitted for App calibrate, waiting on task 1189
parsl.dataflow.dflow DEBUG: Task 1190 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac33d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1190 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1191 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1191 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac3b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1191 has memoization hash 7f0f6df96bec5b7a20a3f5e0286c23b9
parsl.dataflow.memoization INFO: Task 1191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1191 try 0 launched on executor multi with executor id 398
parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2e293971-1040-42e4-b110-64babd4b25c7_isr_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2e293971-1040-42e4-b110-64babd4b25c7_isr_947420_61.stderr
parsl.dataflow.dflow DEBUG: Task 1192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1192 submitted for App characterizeImage, waiting on task 1191
parsl.dataflow.dflow DEBUG: Task 1192 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac37d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1192 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1193 submitted for App calibrate, waiting on task 1192
parsl.dataflow.dflow DEBUG: Task 1193 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad4950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1193 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1194 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1194 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad4f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1194 has memoization hash 4e00c820086aba0dfb9a99ebe5b7a7fd
parsl.dataflow.memoization INFO: Task 1194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1194 try 0 launched on executor multi with executor id 399
parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7927e069-94b0-4ffb-84c9-c4b8ea809d02_isr_947523_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7927e069-94b0-4ffb-84c9-c4b8ea809d02_isr_947523_54.stderr
parsl.dataflow.dflow DEBUG: Task 1195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1195 submitted for App characterizeImage, waiting on task 1194
parsl.dataflow.dflow DEBUG: Task 1195 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad4c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1195 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1196 submitted for App calibrate, waiting on task 1195
parsl.dataflow.dflow DEBUG: Task 1196 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad57d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1196 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1197 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1197 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad6310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1197 has memoization hash 8b458c89bbec99d7735dafdfbc174e78
parsl.dataflow.memoization INFO: Task 1197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Parsl task 1197 try 0 launched on executor multi with executor id 400
parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8e4cd71c-61a3-44d4-a32c-61beeaefb63e_isr_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8e4cd71c-61a3-44d4-a32c-61beeaefb63e_isr_947418_61.stderr
parsl.dataflow.dflow DEBUG: Task 1198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1198 submitted for App characterizeImage, waiting on task 1197
parsl.dataflow.dflow DEBUG: Task 1198 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad6ed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1198 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1199 submitted for App calibrate, waiting on task 1198
parsl.dataflow.dflow DEBUG: Task 1199 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae42d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1199 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1200 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1200 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae45d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1200 has memoization hash 33da2977be7f8920f78aecab98a8a845
parsl.dataflow.memoization INFO: Task 1200 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1200 try 0 launched on executor multi with executor id 401
parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4a8f353f-37f8-40b6-ab38-aed4f2de1744_isr_947528_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4a8f353f-37f8-40b6-ab38-aed4f2de1744_isr_947528_57.stderr
parsl.dataflow.dflow DEBUG: Task 1201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1201 submitted for App characterizeImage, waiting on task 1200
parsl.dataflow.dflow DEBUG: Task 1201 set to pending state with AppFuture: <AppFuture at 0x14a6f8c44c10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1201 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1202 submitted for App calibrate, waiting on task 1201
parsl.dataflow.dflow DEBUG: Task 1202 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae4d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1202 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1203 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1203 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae5990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1203 has memoization hash e7ffb3d4d84d95e12169e3472288da00
parsl.dataflow.memoization INFO: Task 1203 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1203 try 0 launched on executor multi with executor id 402
parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8ad6aee9-9ae9-400d-a466-59c258f09cbb_isr_947420_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8ad6aee9-9ae9-400d-a466-59c258f09cbb_isr_947420_37.stderr
parsl.dataflow.dflow DEBUG: Task 1204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1204 submitted for App characterizeImage, waiting on task 1203
parsl.dataflow.dflow DEBUG: Task 1204 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae5d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1204 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1205 submitted for App calibrate, waiting on task 1204
parsl.dataflow.dflow DEBUG: Task 1205 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae6b50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1205 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1206 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1206 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae6f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1206 has memoization hash 60f66b5d36f6b9805eb34d1a6c81ab10
parsl.dataflow.memoization INFO: Task 1206 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1206 try 0 launched on executor multi with executor id 403
parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/31e7cffa-c4ea-4905-982d-3d97d2b27538_isr_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/31e7cffa-c4ea-4905-982d-3d97d2b27538_isr_947524_61.stderr
parsl.dataflow.dflow DEBUG: Task 1207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1207 submitted for App characterizeImage, waiting on task 1206
parsl.dataflow.dflow DEBUG: Task 1207 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae7090 state=pending>
parsl.dataflow.dflow DEBUG: Task 1207 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1208 submitted for App calibrate, waiting on task 1207
parsl.dataflow.dflow DEBUG: Task 1208 set to pending state with AppFuture: <AppFuture at 0x14a6f8ae7c90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1208 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1209 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1209 set to pending state with AppFuture: <AppFuture at 0x14a6f8aec310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1209 has memoization hash 113fe5f4d787d5fae27b3bf4e092098d
parsl.dataflow.memoization INFO: Task 1209 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1209 try 0 launched on executor multi with executor id 404
parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/416e64e8-9f32-4d6a-a207-2ce3ffa01226_isr_947526_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/416e64e8-9f32-4d6a-a207-2ce3ffa01226_isr_947526_29.stderr
parsl.dataflow.dflow DEBUG: Task 1210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1210 submitted for App characterizeImage, waiting on task 1209
parsl.dataflow.dflow DEBUG: Task 1210 set to pending state with AppFuture: <AppFuture at 0x14a6f8d6e750 state=pending>
parsl.dataflow.dflow DEBUG: Task 1210 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1211 submitted for App calibrate, waiting on task 1210
parsl.dataflow.dflow DEBUG: Task 1211 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad7510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1211 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1212 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1212 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad7950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1212 has memoization hash a127629c80cc1bedc15d429e8f163c66
parsl.dataflow.memoization INFO: Task 1212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1212 try 0 launched on executor multi with executor id 405
parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9e666198-bcef-45fa-a7b1-fd8d6c9b0da0_isr_947524_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9e666198-bcef-45fa-a7b1-fd8d6c9b0da0_isr_947524_59.stderr
parsl.dataflow.dflow DEBUG: Task 1213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1213 submitted for App characterizeImage, waiting on task 1212
parsl.dataflow.dflow DEBUG: Task 1213 set to pending state with AppFuture: <AppFuture at 0x14a6f8ad7650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1213 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1214 submitted for App calibrate, waiting on task 1213
parsl.dataflow.dflow DEBUG: Task 1214 set to pending state with AppFuture: <AppFuture at 0x14a6f8aed190 state=pending>
parsl.dataflow.dflow DEBUG: Task 1214 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1215 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1215 set to pending state with AppFuture: <AppFuture at 0x14a6f8aeda90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1215 has memoization hash f1956afe0f32e43e55331036961e5563
parsl.dataflow.memoization INFO: Task 1215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1215 try 0 launched on executor multi with executor id 406
parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b81ffc02-3e52-4b50-8faf-e260709ce2f9_isr_947418_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b81ffc02-3e52-4b50-8faf-e260709ce2f9_isr_947418_26.stderr
parsl.dataflow.dflow DEBUG: Task 1216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1216 submitted for App characterizeImage, waiting on task 1215
parsl.dataflow.dflow DEBUG: Task 1216 set to pending state with AppFuture: <AppFuture at 0x14a6f8aedbd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1216 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1217 submitted for App calibrate, waiting on task 1216
parsl.dataflow.dflow DEBUG: Task 1217 set to pending state with AppFuture: <AppFuture at 0x14a6f8aee6d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1217 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1218 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1218 set to pending state with AppFuture: <AppFuture at 0x14a6f8aeeed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1218 has memoization hash f3d82093176ae955fcc2e69f750e9621
parsl.dataflow.memoization INFO: Task 1218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1218 try 0 launched on executor multi with executor id 407
parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a1d11a12-05f5-48cd-b2c0-3c077f031733_isr_947526_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a1d11a12-05f5-48cd-b2c0-3c077f031733_isr_947526_46.stderr
parsl.dataflow.dflow DEBUG: Task 1219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1219 submitted for App characterizeImage, waiting on task 1218
parsl.dataflow.dflow DEBUG: Task 1219 set to pending state with AppFuture: <AppFuture at 0x14a6f8aefe90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1219 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1220 submitted for App calibrate, waiting on task 1219
parsl.dataflow.dflow DEBUG: Task 1220 set to pending state with AppFuture: <AppFuture at 0x14a6f8aefed0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1220 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1221 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1221 set to pending state with AppFuture: <AppFuture at 0x14a6f8af4310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1221 has memoization hash 7e5bba796b7eb527905d2d261ad073eb
parsl.dataflow.memoization INFO: Task 1221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1221 try 0 launched on executor multi with executor id 408
parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/012df7b1-6c32-4062-b6c4-7d646ca5bab6_isr_947523_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/012df7b1-6c32-4062-b6c4-7d646ca5bab6_isr_947523_34.stderr
parsl.dataflow.dflow DEBUG: Task 1222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1222 submitted for App characterizeImage, waiting on task 1221
parsl.dataflow.dflow DEBUG: Task 1222 set to pending state with AppFuture: <AppFuture at 0x14a6f8af5090 state=pending>
parsl.dataflow.dflow DEBUG: Task 1222 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1223 submitted for App calibrate, waiting on task 1222
parsl.dataflow.dflow DEBUG: Task 1223 set to pending state with AppFuture: <AppFuture at 0x14a6f8af4b10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1223 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1224 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1224 set to pending state with AppFuture: <AppFuture at 0x14a6f8af5690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1224 has memoization hash fb0f8f3aad088ecaefdeaebef4982017
parsl.dataflow.memoization INFO: Task 1224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1224 try 0 launched on executor multi with executor id 409
parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0cb6ccc-1fda-4af5-9482-95ed1cd8a5ea_isr_947524_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0cb6ccc-1fda-4af5-9482-95ed1cd8a5ea_isr_947524_21.stderr
parsl.dataflow.dflow DEBUG: Task 1225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1225 submitted for App characterizeImage, waiting on task 1224
parsl.dataflow.dflow DEBUG: Task 1225 set to pending state with AppFuture: <AppFuture at 0x14a6f8af5d10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1225 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1226 submitted for App calibrate, waiting on task 1225
parsl.dataflow.dflow DEBUG: Task 1226 set to pending state with AppFuture: <AppFuture at 0x14a6f8af62d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1226 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1227 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1227 set to pending state with AppFuture: <AppFuture at 0x14a6f8af6b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1227 has memoization hash b2d9f750106cf18bc81c9fb41039a337
parsl.dataflow.memoization INFO: Task 1227 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1227 try 0 launched on executor multi with executor id 410
parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1478c42d-6177-4db8-bbcb-f998b66f50ac_isr_947528_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1478c42d-6177-4db8-bbcb-f998b66f50ac_isr_947528_20.stderr
parsl.dataflow.dflow DEBUG: Task 1228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1228 submitted for App characterizeImage, waiting on task 1227
parsl.dataflow.dflow DEBUG: Task 1228 set to pending state with AppFuture: <AppFuture at 0x14a6f8af5fd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1228 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1229 submitted for App calibrate, waiting on task 1228
parsl.dataflow.dflow DEBUG: Task 1229 set to pending state with AppFuture: <AppFuture at 0x14a6f8af7050 state=pending>
parsl.dataflow.dflow DEBUG: Task 1229 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1230 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1230 set to pending state with AppFuture: <AppFuture at 0x14a6f8af7f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1230 has memoization hash ea8f042285f54dd82a31426630af401e
parsl.dataflow.memoization INFO: Task 1230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
2024-03-14 07:25:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17033299
proc_lsst.multi INFO: [multi] job_id 17033299
2024-03-14 07:25:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033299
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033299
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17033299
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-14 07:25:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-14 07:25:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Parsl task 1230 try 0 launched on executor multi with executor id 411
parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/0a0f642d-a9f1-4627-9909-bcb5f2ec6d38_isr_947528_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/0a0f642d-a9f1-4627-9909-bcb5f2ec6d38_isr_947528_46.stderr
parsl.dataflow.dflow DEBUG: Task 1231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1231 submitted for App characterizeImage, waiting on task 1230
parsl.dataflow.dflow DEBUG: Task 1231 set to pending state with AppFuture: <AppFuture at 0x14a6f8ab74d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1231 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1232 submitted for App calibrate, waiting on task 1231
parsl.dataflow.dflow DEBUG: Task 1232 set to pending state with AppFuture: <AppFuture at 0x14a6f8af85d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1232 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1233 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1233 set to pending state with AppFuture: <AppFuture at 0x14a6f8af8e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1233 has memoization hash b74ebbd09386ac7b2355882da62712a8
parsl.dataflow.memoization INFO: Task 1233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1233 try 0 launched on executor multi with executor id 412
parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3c1b9b5-1f32-446d-addd-24a1b8b21ae7_isr_947523_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3c1b9b5-1f32-446d-addd-24a1b8b21ae7_isr_947523_18.stderr
parsl.dataflow.dflow DEBUG: Task 1234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1234 submitted for App characterizeImage, waiting on task 1233
parsl.dataflow.dflow DEBUG: Task 1234 set to pending state with AppFuture: <AppFuture at 0x14a6f8af9010 state=pending>
parsl.dataflow.dflow DEBUG: Task 1234 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1235 submitted for App calibrate, waiting on task 1234
parsl.dataflow.dflow DEBUG: Task 1235 set to pending state with AppFuture: <AppFuture at 0x14a6f8af9710 state=pending>
parsl.dataflow.dflow DEBUG: Task 1235 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1236 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1236 set to pending state with AppFuture: <AppFuture at 0x14a6f8afa3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1236 has memoization hash aa7738ce040d309efdeaec5bf777287c
parsl.dataflow.memoization INFO: Task 1236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1236 try 0 launched on executor multi with executor id 413
parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5463bb8b-168c-4a77-82b0-398be0be8ab7_isr_947524_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5463bb8b-168c-4a77-82b0-398be0be8ab7_isr_947524_32.stderr
parsl.dataflow.dflow DEBUG: Task 1237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1237 submitted for App characterizeImage, waiting on task 1236
parsl.dataflow.dflow DEBUG: Task 1237 set to pending state with AppFuture: <AppFuture at 0x14a6f8af9650 state=pending>
parsl.dataflow.dflow DEBUG: Task 1237 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1238 submitted for App calibrate, waiting on task 1237
parsl.dataflow.dflow DEBUG: Task 1238 set to pending state with AppFuture: <AppFuture at 0x14a6f8afaa50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1238 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1239 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1239 set to pending state with AppFuture: <AppFuture at 0x14a6f8afb710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1239 has memoization hash f16f6be98ebefcf5b45fd028ded7ed71
parsl.dataflow.memoization INFO: Task 1239 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1239 try 0 launched on executor multi with executor id 414
parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9fc5494a-bce4-4f17-9800-3e01d9bfe5ae_isr_947523_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9fc5494a-bce4-4f17-9800-3e01d9bfe5ae_isr_947523_3.stderr
parsl.dataflow.dflow DEBUG: Task 1240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1240 submitted for App characterizeImage, waiting on task 1239
parsl.dataflow.dflow DEBUG: Task 1240 set to pending state with AppFuture: <AppFuture at 0x14a6f8afbfd0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1240 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1241 submitted for App calibrate, waiting on task 1240
parsl.dataflow.dflow DEBUG: Task 1241 set to pending state with AppFuture: <AppFuture at 0x14a6f8b085d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1241 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1242 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1242 set to pending state with AppFuture: <AppFuture at 0x14a6f8b08bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1242 has memoization hash b4e2e073f00062dfbf31bf6bf2dc06be
parsl.dataflow.memoization INFO: Task 1242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1242 try 0 launched on executor multi with executor id 415
parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/831b7eb8-e38d-482c-bc13-44469ffd5d1f_isr_947528_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/831b7eb8-e38d-482c-bc13-44469ffd5d1f_isr_947528_43.stderr
parsl.dataflow.dflow DEBUG: Task 1243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1243 submitted for App characterizeImage, waiting on task 1242
parsl.dataflow.dflow DEBUG: Task 1243 set to pending state with AppFuture: <AppFuture at 0x14a6f8ac0410 state=pending>
parsl.dataflow.dflow DEBUG: Task 1243 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1244 submitted for App calibrate, waiting on task 1243
parsl.dataflow.dflow DEBUG: Task 1244 set to pending state with AppFuture: <AppFuture at 0x14a6f8b09a90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1244 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1245 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1245 set to pending state with AppFuture: <AppFuture at 0x14a6f8b09fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1245 has memoization hash 87dd1035918938c05fa1323d16e6ab5a
parsl.dataflow.memoization INFO: Task 1245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1245 try 0 launched on executor multi with executor id 416
parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/03086161-c6d2-49fd-a0eb-05696ff46ec3_isr_947418_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/03086161-c6d2-49fd-a0eb-05696ff46ec3_isr_947418_28.stderr
parsl.dataflow.dflow DEBUG: Task 1246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1246 submitted for App characterizeImage, waiting on task 1245
parsl.dataflow.dflow DEBUG: Task 1246 set to pending state with AppFuture: <AppFuture at 0x14a6f8b0aa10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1246 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1247 submitted for App calibrate, waiting on task 1246
parsl.dataflow.dflow DEBUG: Task 1247 set to pending state with AppFuture: <AppFuture at 0x14a6f8b0a990 state=pending>
parsl.dataflow.dflow DEBUG: Task 1247 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1248 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1248 set to pending state with AppFuture: <AppFuture at 0x14a6f8b0b410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1248 has memoization hash 401b23d95415b756d585ae4982bef4db
parsl.dataflow.memoization INFO: Task 1248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1248 try 0 launched on executor multi with executor id 417
parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/415731bd-0501-419d-b8f9-152ab68f51bf_isr_947418_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/415731bd-0501-419d-b8f9-152ab68f51bf_isr_947418_37.stderr
parsl.dataflow.dflow DEBUG: Task 1249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1249 submitted for App characterizeImage, waiting on task 1248
parsl.dataflow.dflow DEBUG: Task 1249 set to pending state with AppFuture: <AppFuture at 0x14a6f8b0b950 state=pending>
parsl.dataflow.dflow DEBUG: Task 1249 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1250 submitted for App calibrate, waiting on task 1249
parsl.dataflow.dflow DEBUG: Task 1250 set to pending state with AppFuture: <AppFuture at 0x14a6f8af9510 state=pending>
parsl.dataflow.dflow DEBUG: Task 1250 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1251 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1251 set to pending state with AppFuture: <AppFuture at 0x14a6f8b18910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1251 has memoization hash fe003e5faadcdfd51f01e4c4eddab595
parsl.dataflow.memoization INFO: Task 1251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1251 try 0 launched on executor multi with executor id 418
parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/508548d2-58d7-4de6-a502-135bb6bdcfaf_isr_947524_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/508548d2-58d7-4de6-a502-135bb6bdcfaf_isr_947524_50.stderr
parsl.dataflow.dflow DEBUG: Task 1252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1252 submitted for App characterizeImage, waiting on task 1251
parsl.dataflow.dflow DEBUG: Task 1252 set to pending state with AppFuture: <AppFuture at 0x14a6f8b099d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1252 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1253 submitted for App calibrate, waiting on task 1252
parsl.dataflow.dflow DEBUG: Task 1253 set to pending state with AppFuture: <AppFuture at 0x14a6f8b19110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1253 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1254 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1254 set to pending state with AppFuture: <AppFuture at 0x14a6f8b19dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1254 has memoization hash afdb2831a93973b5165a5702eaeb4c34
parsl.dataflow.memoization INFO: Task 1254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1254 try 0 launched on executor multi with executor id 419
parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e742706-1830-4a2e-9683-c8e63e61ad7c_isr_947524_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e742706-1830-4a2e-9683-c8e63e61ad7c_isr_947524_39.stderr
parsl.dataflow.dflow DEBUG: Task 1255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1255 submitted for App characterizeImage, waiting on task 1254
parsl.dataflow.dflow DEBUG: Task 1255 set to pending state with AppFuture: <AppFuture at 0x14a6f8b19150 state=pending>
parsl.dataflow.dflow DEBUG: Task 1255 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1256 submitted for App calibrate, waiting on task 1255
parsl.dataflow.dflow DEBUG: Task 1256 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1af10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1256 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1257 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1257 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1b250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1257 has memoization hash c23c8f5c58a2867cf728d5276a1b9d5f
parsl.dataflow.memoization INFO: Task 1257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1257 try 0 launched on executor multi with executor id 420
parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/56896d40-6cfd-4a95-8e7d-3e671c53ca3e_isr_947422_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/56896d40-6cfd-4a95-8e7d-3e671c53ca3e_isr_947422_42.stderr
parsl.dataflow.dflow DEBUG: Task 1258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1258 submitted for App characterizeImage, waiting on task 1257
parsl.dataflow.dflow DEBUG: Task 1258 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1b550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1258 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1259 submitted for App calibrate, waiting on task 1258
parsl.dataflow.dflow DEBUG: Task 1259 set to pending state with AppFuture: <AppFuture at 0x14a6f8c93e90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1259 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1260 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1260 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1c690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1260 has memoization hash 3cf6e8cc4ee6ed76e82a233c7e52cba4
parsl.dataflow.memoization INFO: Task 1260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1260 try 0 launched on executor multi with executor id 421
parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a4676c6-69ee-4a4d-ab7a-cbedc637ec8b_isr_947524_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a4676c6-69ee-4a4d-ab7a-cbedc637ec8b_isr_947524_52.stderr
parsl.dataflow.dflow DEBUG: Task 1261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1261 submitted for App characterizeImage, waiting on task 1260
parsl.dataflow.dflow DEBUG: Task 1261 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1ba10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1261 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1262 submitted for App calibrate, waiting on task 1261
parsl.dataflow.dflow DEBUG: Task 1262 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1d550 state=pending>
parsl.dataflow.dflow DEBUG: Task 1262 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1263 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1263 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1da50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1263 has memoization hash 3774f1cd012c028edfb57a926b6184ac
parsl.dataflow.memoization INFO: Task 1263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1263 try 0 launched on executor multi with executor id 422
parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/379bb84a-62cb-4716-a94e-a4901afcc656_isr_947422_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/379bb84a-62cb-4716-a94e-a4901afcc656_isr_947422_52.stderr
parsl.dataflow.dflow DEBUG: Task 1264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1264 submitted for App characterizeImage, waiting on task 1263
parsl.dataflow.dflow DEBUG: Task 1264 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1ced0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1264 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1265 submitted for App calibrate, waiting on task 1264
parsl.dataflow.dflow DEBUG: Task 1265 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1eb90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1265 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1266 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1266 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1eed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1266 has memoization hash b8152dc0270cd29adc7ea9a9c8e86af8
parsl.dataflow.memoization INFO: Task 1266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1266 try 0 launched on executor multi with executor id 423
parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9877bae2-6764-42c7-b66a-f309977c88df_isr_947528_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9877bae2-6764-42c7-b66a-f309977c88df_isr_947528_48.stderr
parsl.dataflow.dflow DEBUG: Task 1267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1267 submitted for App characterizeImage, waiting on task 1266
parsl.dataflow.dflow DEBUG: Task 1267 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1e390 state=pending>
parsl.dataflow.dflow DEBUG: Task 1267 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1268 submitted for App calibrate, waiting on task 1267
parsl.dataflow.dflow DEBUG: Task 1268 set to pending state with AppFuture: <AppFuture at 0x14a6f8b1fd90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1268 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1269 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1269 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2c350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1269 has memoization hash 14c35b8f8d8f26d799d611d633ba3190
parsl.dataflow.memoization INFO: Task 1269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1269 try 0 launched on executor multi with executor id 424
parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9660d08-1d46-44cb-b3a8-f6085b79b8b4_isr_947422_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9660d08-1d46-44cb-b3a8-f6085b79b8b4_isr_947422_23.stderr
parsl.dataflow.dflow DEBUG: Task 1270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1270 submitted for App characterizeImage, waiting on task 1269
parsl.dataflow.dflow DEBUG: Task 1270 set to pending state with AppFuture: <AppFuture at 0x14a6f8aecb50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1270 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1271 submitted for App calibrate, waiting on task 1270
parsl.dataflow.dflow DEBUG: Task 1271 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2d210 state=pending>
parsl.dataflow.dflow DEBUG: Task 1271 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1272 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1272 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2d750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1272 has memoization hash 435fe5565b542d74cc65fd720eb78107
parsl.dataflow.memoization INFO: Task 1272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1272 try 0 launched on executor multi with executor id 425
parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4932df2d-563f-470f-9875-518264dd86f8_isr_947528_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4932df2d-563f-470f-9875-518264dd86f8_isr_947528_5.stderr
parsl.dataflow.dflow DEBUG: Task 1273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1273 submitted for App characterizeImage, waiting on task 1272
parsl.dataflow.dflow DEBUG: Task 1273 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2d390 state=pending>
parsl.dataflow.dflow DEBUG: Task 1273 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1274 submitted for App calibrate, waiting on task 1273
parsl.dataflow.dflow DEBUG: Task 1274 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2e5d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1274 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1275 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1275 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2ecd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1275 has memoization hash 0929be041cb86dbfc53751d3da97373d
parsl.dataflow.memoization INFO: Task 1275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1275 try 0 launched on executor multi with executor id 426
parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6865c994-e7fd-4771-ba08-bc47c99f43fb_isr_947422_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6865c994-e7fd-4771-ba08-bc47c99f43fb_isr_947422_30.stderr
parsl.dataflow.dflow DEBUG: Task 1276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1276 submitted for App characterizeImage, waiting on task 1275
parsl.dataflow.dflow DEBUG: Task 1276 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2df50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1276 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1277 submitted for App calibrate, waiting on task 1276
parsl.dataflow.dflow DEBUG: Task 1277 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2fad0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1277 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1278 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1278 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2fe50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1278 has memoization hash eaa7f5b90f5a9a146356406769ba731a
parsl.dataflow.memoization INFO: Task 1278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1278 try 0 launched on executor multi with executor id 427
parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5fb6c8b2-f0f1-4fa8-9b8c-48f6112a624b_isr_947420_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5fb6c8b2-f0f1-4fa8-9b8c-48f6112a624b_isr_947420_44.stderr
parsl.dataflow.dflow DEBUG: Task 1279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1279 submitted for App characterizeImage, waiting on task 1278
parsl.dataflow.dflow DEBUG: Task 1279 set to pending state with AppFuture: <AppFuture at 0x14a6f8b2f590 state=pending>
parsl.dataflow.dflow DEBUG: Task 1279 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1280 submitted for App calibrate, waiting on task 1279
parsl.dataflow.dflow DEBUG: Task 1280 set to pending state with AppFuture: <AppFuture at 0x14a6f8b38f90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1280 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1281 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1281 set to pending state with AppFuture: <AppFuture at 0x14a6f8b394d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1281 has memoization hash 59ecb3d5bc483328b22ba132d5b6b538
parsl.dataflow.memoization INFO: Task 1281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1281 try 0 launched on executor multi with executor id 428
parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/327cc6a0-e056-4973-bdc5-ae38ae661419_isr_947523_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/327cc6a0-e056-4973-bdc5-ae38ae661419_isr_947523_25.stderr
parsl.dataflow.dflow DEBUG: Task 1282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1282 submitted for App characterizeImage, waiting on task 1281
parsl.dataflow.dflow DEBUG: Task 1282 set to pending state with AppFuture: <AppFuture at 0x14a6f8d69050 state=pending>
parsl.dataflow.dflow DEBUG: Task 1282 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1283 submitted for App calibrate, waiting on task 1282
parsl.dataflow.dflow DEBUG: Task 1283 set to pending state with AppFuture: <AppFuture at 0x14a6f8b39c50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1283 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1284 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1284 set to pending state with AppFuture: <AppFuture at 0x14a6f8b3a890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1284 has memoization hash 2b4df1c179d02037fa3f576e560c16bb
parsl.dataflow.memoization INFO: Task 1284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1284 try 0 launched on executor multi with executor id 429
parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b8688102-fe1c-4428-bf7e-b59f1af19e5a_isr_947526_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b8688102-fe1c-4428-bf7e-b59f1af19e5a_isr_947526_1.stderr
parsl.dataflow.dflow DEBUG: Task 1285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1285 submitted for App characterizeImage, waiting on task 1284
parsl.dataflow.dflow DEBUG: Task 1285 set to pending state with AppFuture: <AppFuture at 0x14a6f8b389d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1285 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1286 submitted for App calibrate, waiting on task 1285
parsl.dataflow.dflow DEBUG: Task 1286 set to pending state with AppFuture: <AppFuture at 0x14a6f8b3b2d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1286 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1287 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1287 set to pending state with AppFuture: <AppFuture at 0x14a6f8b3bd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1287 has memoization hash 0811b4a6ba1f9bed9e23230c9a342a34
parsl.dataflow.memoization INFO: Task 1287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1287 try 0 launched on executor multi with executor id 430
parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0e242350-6191-48b7-949f-c6e0f3f83a14_isr_947523_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0e242350-6191-48b7-949f-c6e0f3f83a14_isr_947523_14.stderr
parsl.dataflow.dflow DEBUG: Task 1288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1288 submitted for App characterizeImage, waiting on task 1287
parsl.dataflow.dflow DEBUG: Task 1288 set to pending state with AppFuture: <AppFuture at 0x14a6f8b38810 state=pending>
parsl.dataflow.dflow DEBUG: Task 1288 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1289 submitted for App calibrate, waiting on task 1288
parsl.dataflow.dflow DEBUG: Task 1289 set to pending state with AppFuture: <AppFuture at 0x14a6f8b44610 state=pending>
parsl.dataflow.dflow DEBUG: Task 1289 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1290 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1290 set to pending state with AppFuture: <AppFuture at 0x14a6f8b45090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1290 has memoization hash 28ed83bc08aeb106eb2faf7b91f663bb
parsl.dataflow.memoization INFO: Task 1290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1290 try 0 launched on executor multi with executor id 431
parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/321cd90e-0530-4e0a-9bd0-b7e79e6d6f77_isr_947524_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/321cd90e-0530-4e0a-9bd0-b7e79e6d6f77_isr_947524_11.stderr
parsl.dataflow.dflow DEBUG: Task 1291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1291 submitted for App characterizeImage, waiting on task 1290
parsl.dataflow.dflow DEBUG: Task 1291 set to pending state with AppFuture: <AppFuture at 0x14a6f8b46110 state=pending>
parsl.dataflow.dflow DEBUG: Task 1291 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1292 submitted for App calibrate, waiting on task 1291
parsl.dataflow.dflow DEBUG: Task 1292 set to pending state with AppFuture: <AppFuture at 0x14a6f8b45f10 state=pending>
parsl.dataflow.dflow DEBUG: Task 1292 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1293 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1293 set to pending state with AppFuture: <AppFuture at 0x14a6f8b46510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1293 has memoization hash 7c9b3c71ea3a0c36840a0f4b7e3c6116
parsl.dataflow.memoization INFO: Task 1293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1293 try 0 launched on executor multi with executor id 432
parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/db247818-917a-4e58-a4ef-b158bba56cec_isr_947528_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/db247818-917a-4e58-a4ef-b158bba56cec_isr_947528_41.stderr
parsl.dataflow.dflow DEBUG: Task 1294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1294 submitted for App characterizeImage, waiting on task 1293
parsl.dataflow.dflow DEBUG: Task 1294 set to pending state with AppFuture: <AppFuture at 0x14a6f8b45890 state=pending>
parsl.dataflow.dflow DEBUG: Task 1294 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1295 submitted for App calibrate, waiting on task 1294
parsl.dataflow.dflow DEBUG: Task 1295 set to pending state with AppFuture: <AppFuture at 0x14a6f8b46d90 state=pending>
parsl.dataflow.dflow DEBUG: Task 1295 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1296 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1296 set to pending state with AppFuture: <AppFuture at 0x14a6f8b47990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1296 has memoization hash 1d7f1fb8ed553fc6f256eac103cbaa72
parsl.dataflow.memoization INFO: Task 1296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1296 try 0 launched on executor multi with executor id 433
parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6eba221a-d014-4720-82eb-1d8e8dea66d5_isr_947523_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6eba221a-d014-4720-82eb-1d8e8dea66d5_isr_947523_43.stderr
parsl.dataflow.dflow DEBUG: Task 1297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1297 submitted for App characterizeImage, waiting on task 1296
parsl.dataflow.dflow DEBUG: Task 1297 set to pending state with AppFuture: <AppFuture at 0x14a6f8b47d50 state=pending>
parsl.dataflow.dflow DEBUG: Task 1297 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1298 submitted for App calibrate, waiting on task 1297
parsl.dataflow.dflow DEBUG: Task 1298 set to pending state with AppFuture: <AppFuture at 0x14a6f8b4c690 state=pending>
parsl.dataflow.dflow DEBUG: Task 1298 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1299 submitted for App isr, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1299 set to pending state with AppFuture: <AppFuture at 0x14a6f8b4ce10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1299 has memoization hash 515c68fef976624118c6b966bf3237cf
parsl.dataflow.memoization INFO: Task 1299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1299 try 0 launched on executor multi with executor id 434
parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2e0e3ed0-6f13-452d-bb5d-b5a7dde87686_isr_947418_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2e0e3ed0-6f13-452d-bb5d-b5a7dde87686_isr_947418_38.stderr
parsl.dataflow.dflow DEBUG: Task 1300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1300 submitted for App characterizeImage, waiting on task 1299
parsl.dataflow.dflow DEBUG: Task 1300 set to pending state with AppFuture: <AppFuture at 0x14a6f8b4c410 state=pending>
parsl.dataflow.dflow DEBUG: Task 1300 has outstanding dependencies, so launch_if_ready skipping
parsl.dataflow.dflow DEBUG: Task 1301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1301 submitted for App calibrate, waiting on task 1300
parsl.dataflow.dflow DEBUG: Task 1301 set to pending state with AppFuture: <AppFuture at 0x14a6f8b4d7d0 state=pending>
parsl.dataflow.dflow DEBUG: Task 1301 has outstanding dependencies, so launch_if_ready skipping
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17033300
proc_lsst.multi INFO: [multi] job_id 17033300
2024-03-14 07:25:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033300
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033300
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17033300
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-14 07:25:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-14 07:25:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17033301
proc_lsst.multi INFO: [multi] job_id 17033301
2024-03-14 07:25:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033301
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033301
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17033301
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-14 07:25:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-14 07:25:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17033302
proc_lsst.multi INFO: [multi] job_id 17033302
2024-03-14 07:25:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033302
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033302
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17033302
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-14 07:25:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-14 07:25:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17033303
proc_lsst.multi INFO: [multi] job_id 17033303
2024-03-14 07:25:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033303
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033303
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17033303
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-14 07:25:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-14 07:25:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17033304
proc_lsst.multi INFO: [multi] job_id 17033304
2024-03-14 07:25:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 07:25:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033304
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033304
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17033304
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-14 07:25:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-14 07:25:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 07:25:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17033305
proc_lsst.multi INFO: [multi] job_id 17033305
2024-03-14 07:25:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 07:25:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033305
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033305
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17033305
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-14 07:25:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-14 07:25:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 07:25:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17033306
proc_lsst.multi INFO: [multi] job_id 17033306
2024-03-14 07:25:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 07:25:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033306
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033306
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17033306
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-14 07:25:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-14 07:25:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 07:25:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17033307
proc_lsst.multi INFO: [multi] job_id 17033307
2024-03-14 07:25:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 07:25:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033307
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033307
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17033307
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-14 07:25:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-14 07:25:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 07:25:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17033308
proc_lsst.multi INFO: [multi] job_id 17033308
2024-03-14 07:25:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 07:25:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033308
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033308
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17033308
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-14 07:25:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-14 07:25:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 07:25:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17033309
proc_lsst.multi INFO: [multi] job_id 17033309
2024-03-14 07:25:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 07:25:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:45 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033309
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033309
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17033309
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-14 07:25:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-14 07:25:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 07:25:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17033310
proc_lsst.multi INFO: [multi] job_id 17033310
2024-03-14 07:25:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 07:25:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:45 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033310
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033310
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17033310
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-14 07:25:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-14 07:25:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 07:25:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17033311
proc_lsst.multi INFO: [multi] job_id 17033311
2024-03-14 07:25:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 07:25:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:45 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033311
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033311
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17033311
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-14 07:25:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-14 07:25:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 07:25:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17033312
proc_lsst.multi INFO: [multi] job_id 17033312
2024-03-14 07:25:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 07:25:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:46 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033312
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033312
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17033312
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-14 07:25:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-14 07:25:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 07:25:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17033313
proc_lsst.multi INFO: [multi] job_id 17033313
2024-03-14 07:25:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 07:25:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:46 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033313
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033313
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17033313
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-14 07:25:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-14 07:25:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 07:25:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17033314
proc_lsst.multi INFO: [multi] job_id 17033314
2024-03-14 07:25:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 07:25:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:46 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033314
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033314
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17033314
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-14 07:25:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-14 07:25:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 07:25:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17033315
proc_lsst.multi INFO: [multi] job_id 17033315
2024-03-14 07:25:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 07:25:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033315
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033315
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17033315
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-14 07:25:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17033316
proc_lsst.multi INFO: [multi] job_id 17033316
2024-03-14 07:25:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 07:25:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033316
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033316
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17033316
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-14 07:25:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17033317
proc_lsst.multi INFO: [multi] job_id 17033317
2024-03-14 07:25:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 07:25:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033317
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033317
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17033317
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-14 07:25:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17033318
proc_lsst.multi INFO: [multi] job_id 17033318
2024-03-14 07:25:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033318
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033318
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17033318
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-14 07:25:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17033319
proc_lsst.multi INFO: [multi] job_id 17033319
2024-03-14 07:25:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-14 07:25:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033319
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033319
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17033319
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-14 07:25:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-14 07:25:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17033320
proc_lsst.multi INFO: [multi] job_id 17033320
2024-03-14 07:25:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-14 07:25:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033320
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033320
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17033320
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-14 07:25:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17033321
proc_lsst.multi INFO: [multi] job_id 17033321
2024-03-14 07:25:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-14 07:25:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033321
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033321
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17033321
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-14 07:25:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17033322
proc_lsst.multi INFO: [multi] job_id 17033322
2024-03-14 07:25:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-14 07:25:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033322
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033322
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17033322
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-14 07:25:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17033323
proc_lsst.multi INFO: [multi] job_id 17033323
2024-03-14 07:25:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-14 07:25:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033323
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033323
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17033323
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-14 07:25:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17033324
proc_lsst.multi INFO: [multi] job_id 17033324
2024-03-14 07:25:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-14 07:25:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033324
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033324
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17033324
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-14 07:25:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-14 07:25:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17033325
proc_lsst.multi INFO: [multi] job_id 17033325
2024-03-14 07:25:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-14 07:25:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033325
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033325
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17033325
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-14 07:25:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17033326
proc_lsst.multi INFO: [multi] job_id 17033326
2024-03-14 07:25:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-14 07:25:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033326
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033326
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17033326
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-14 07:25:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17033327
proc_lsst.multi INFO: [multi] job_id 17033327
2024-03-14 07:25:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-14 07:25:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033327
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033327
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17033327
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-14 07:25:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17033328
proc_lsst.multi INFO: [multi] job_id 17033328
2024-03-14 07:25:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17033328
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17033328
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17033328
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-14 07:25:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17033329
proc_lsst.multi INFO: [multi] job_id 17033329
2024-03-14 07:25:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:49 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17033329
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17033329
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17033329
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-14 07:25:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033330
proc_lsst.multi INFO: [multi] job_id 17033330
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17033330
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17033330
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17033330
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033331
proc_lsst.multi INFO: [multi] job_id 17033331
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17033331
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17033331
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17033331
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033332
proc_lsst.multi INFO: [multi] job_id 17033332
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17033332
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17033332
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17033332
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033333
proc_lsst.multi INFO: [multi] job_id 17033333
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17033333
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17033333
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17033333
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033334
proc_lsst.multi INFO: [multi] job_id 17033334
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033334
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033334
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17033334
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033335
proc_lsst.multi INFO: [multi] job_id 17033335
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033335
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033335
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17033335
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033336
proc_lsst.multi INFO: [multi] job_id 17033336
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033336
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033336
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17033336
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17033337
proc_lsst.multi INFO: [multi] job_id 17033337
2024-03-14 07:25:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033337
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033337
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17033337
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-14 07:25:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17033338
proc_lsst.multi INFO: [multi] job_id 17033338
2024-03-14 07:25:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033338
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033338
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17033338
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-14 07:25:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17033339
proc_lsst.multi INFO: [multi] job_id 17033339
2024-03-14 07:25:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 07:25:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033339
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033339
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17033339
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-14 07:25:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17033340
proc_lsst.multi INFO: [multi] job_id 17033340
2024-03-14 07:25:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 07:25:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033340
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033340
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17033340
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-14 07:25:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17033341
proc_lsst.multi INFO: [multi] job_id 17033341
2024-03-14 07:25:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 07:25:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033341
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033341
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17033341
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-14 07:25:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 07:25:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17033342
proc_lsst.multi INFO: [multi] job_id 17033342
2024-03-14 07:25:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 07:25:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:52 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033342
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033342
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17033342
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-14 07:25:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17033343
proc_lsst.multi INFO: [multi] job_id 17033343
2024-03-14 07:25:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 07:25:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:52 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033343
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033343
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17033343
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 434
parsl.dataflow.strategy DEBUG: Executor multi has 434 active tasks, 1/45 running/pending blocks, and 15 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 46 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 63 more blocks
parsl.executors.status_handling INFO: Scaling out by 63 blocks
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-14 07:25:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17033344
proc_lsst.multi INFO: [multi] job_id 17033344
2024-03-14 07:25:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 07:25:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:52 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033344
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033344
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17033344
parsl.executors.status_handling INFO: Allocated block ID 47
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47
2024-03-14 07:25:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 07:25:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17033345
proc_lsst.multi INFO: [multi] job_id 17033345
2024-03-14 07:25:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 07:25:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:53 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033345
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033345
parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17033345
parsl.executors.status_handling INFO: Allocated block ID 48
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48
2024-03-14 07:25:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17033346
proc_lsst.multi INFO: [multi] job_id 17033346
2024-03-14 07:25:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 07:25:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:53 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033346
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033346
parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17033346
parsl.executors.status_handling INFO: Allocated block ID 49
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49
2024-03-14 07:25:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17033347
proc_lsst.multi INFO: [multi] job_id 17033347
2024-03-14 07:25:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 07:25:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:53 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033347
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033347
parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17033347
parsl.executors.status_handling INFO: Allocated block ID 50
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50
2024-03-14 07:25:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 07:25:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17033348
proc_lsst.multi INFO: [multi] job_id 17033348
2024-03-14 07:25:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 07:25:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033348
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033348
parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17033348
parsl.executors.status_handling INFO: Allocated block ID 51
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51
2024-03-14 07:25:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17033349
proc_lsst.multi INFO: [multi] job_id 17033349
2024-03-14 07:25:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 07:25:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033349
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033349
parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17033349
parsl.executors.status_handling INFO: Allocated block ID 52
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52
2024-03-14 07:25:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17033350
proc_lsst.multi INFO: [multi] job_id 17033350
2024-03-14 07:25:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 07:25:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033350
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033350
parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17033350
parsl.executors.status_handling INFO: Allocated block ID 53
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53
2024-03-14 07:25:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17033351
proc_lsst.multi INFO: [multi] job_id 17033351
2024-03-14 07:25:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 07:25:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033351
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033351
parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17033351
parsl.executors.status_handling INFO: Allocated block ID 54
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54
2024-03-14 07:25:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17033352
proc_lsst.multi INFO: [multi] job_id 17033352
2024-03-14 07:25:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 07:25:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033352
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033352
parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17033352
parsl.executors.status_handling INFO: Allocated block ID 55
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55
2024-03-14 07:25:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 07:25:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17033353
proc_lsst.multi INFO: [multi] job_id 17033353
2024-03-14 07:25:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:55 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17033353
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17033353
parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17033353
parsl.executors.status_handling INFO: Allocated block ID 56
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56
2024-03-14 07:25:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
2024-03-14 07:25:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:55 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 07:25:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17033354
proc_lsst.multi INFO: [multi] job_id 17033354
2024-03-14 07:25:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:56 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033354
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033354
parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17033354
parsl.executors.status_handling INFO: Allocated block ID 57
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57
2024-03-14 07:25:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
2024-03-14 07:25:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:56 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:57 proc_lsst.multi:170 [INFO]  [multi] job_id 17033355
proc_lsst.multi INFO: [multi] job_id 17033355
2024-03-14 07:25:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:57 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033355
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033355
parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17033355
parsl.executors.status_handling INFO: Allocated block ID 58
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58
2024-03-14 07:25:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:57 proc_lsst.multi:170 [INFO]  [multi] job_id 17033356
proc_lsst.multi INFO: [multi] job_id 17033356
2024-03-14 07:25:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:57 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033356
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033356
parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17033356
parsl.executors.status_handling INFO: Allocated block ID 59
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59
2024-03-14 07:25:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:57 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 07:25:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:58 proc_lsst.multi:170 [INFO]  [multi] job_id 17033357
proc_lsst.multi INFO: [multi] job_id 17033357
2024-03-14 07:25:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:58 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033357
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033357
parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17033357
parsl.executors.status_handling INFO: Allocated block ID 60
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60
2024-03-14 07:25:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
2024-03-14 07:25:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:58 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 07:25:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17033358
proc_lsst.multi INFO: [multi] job_id 17033358
2024-03-14 07:25:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033358
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033358
parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17033358
parsl.executors.status_handling INFO: Allocated block ID 61
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61
2024-03-14 07:25:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17033359
proc_lsst.multi INFO: [multi] job_id 17033359
2024-03-14 07:25:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 07:25:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033359
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033359
parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17033359
parsl.executors.status_handling INFO: Allocated block ID 62
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62
2024-03-14 07:25:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17033360
proc_lsst.multi INFO: [multi] job_id 17033360
2024-03-14 07:25:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 07:25:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033360
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033360
parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17033360
parsl.executors.status_handling INFO: Allocated block ID 63
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63
2024-03-14 07:25:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17033361
proc_lsst.multi INFO: [multi] job_id 17033361
2024-03-14 07:25:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 07:25:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033361
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033361
parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17033361
parsl.executors.status_handling INFO: Allocated block ID 64
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64
2024-03-14 07:25:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:25:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17033362
proc_lsst.multi INFO: [multi] job_id 17033362
2024-03-14 07:25:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 07:25:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033362
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033362
parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17033362
parsl.executors.status_handling INFO: Allocated block ID 65
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65
2024-03-14 07:25:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:25:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:25:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 07:25:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:25:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 230d18e67c83cb96c6b5dc5deb994075 with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 47e3502e0d47b70cc160c0a9f368419b
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 435
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b06a3720-a39e-429b-940f-c6c8bc60ccdd_characterizeImage_947526_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b06a3720-a39e-429b-940f-c6c8bc60ccdd_characterizeImage_947526_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a0b527bb-645b-4d38-a92b-89247da6b626_isr_947526_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a0b527bb-645b-4d38-a92b-89247da6b626_isr_947526_14.stderr
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31efa13366746faea989734dfe566764 with result from task 0
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 706006c067df72bc2ef9fbe4fa80e3fd
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4663e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 436
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cedd7a7a-ed9a-4390-9bfb-cbaa5c3907be_characterizeImage_947524_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cedd7a7a-ed9a-4390-9bfb-cbaa5c3907be_characterizeImage_947524_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dc3860e4-1579-45f9-a024-9a194b98b005_isr_947524_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dc3860e4-1579-45f9-a024-9a194b98b005_isr_947524_31.stderr
2024-03-14 07:26:00 proc_lsst.multi:170 [INFO]  [multi] job_id 17033363
proc_lsst.multi INFO: [multi] job_id 17033363
2024-03-14 07:26:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 07:26:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:00 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033363
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033363
parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17033363
parsl.executors.status_handling INFO: Allocated block ID 66
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66
2024-03-14 07:26:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:00 proc_lsst.multi:170 [INFO]  [multi] job_id 17033364
proc_lsst.multi INFO: [multi] job_id 17033364
2024-03-14 07:26:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 07:26:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:00 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033364
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033364
parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17033364
parsl.executors.status_handling INFO: Allocated block ID 67
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67
2024-03-14 07:26:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:00 proc_lsst.multi:170 [INFO]  [multi] job_id 17033365
proc_lsst.multi INFO: [multi] job_id 17033365
2024-03-14 07:26:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 07:26:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:00 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033365
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033365
parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17033365
parsl.executors.status_handling INFO: Allocated block ID 68
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68
2024-03-14 07:26:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:00 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 07:26:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:01 proc_lsst.multi:170 [INFO]  [multi] job_id 17033366
proc_lsst.multi INFO: [multi] job_id 17033366
2024-03-14 07:26:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 07:26:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:01 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033366
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033366
parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17033366
parsl.executors.status_handling INFO: Allocated block ID 69
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69
2024-03-14 07:26:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:01 proc_lsst.multi:170 [INFO]  [multi] job_id 17033367
proc_lsst.multi INFO: [multi] job_id 17033367
2024-03-14 07:26:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 07:26:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:01 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033367
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033367
parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17033367
parsl.executors.status_handling INFO: Allocated block ID 70
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70
2024-03-14 07:26:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:01 proc_lsst.multi:170 [INFO]  [multi] job_id 17033368
proc_lsst.multi INFO: [multi] job_id 17033368
2024-03-14 07:26:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 07:26:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:01 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033368
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033368
parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17033368
parsl.executors.status_handling INFO: Allocated block ID 71
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71
2024-03-14 07:26:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:01 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 07:26:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:02 proc_lsst.multi:170 [INFO]  [multi] job_id 17033369
proc_lsst.multi INFO: [multi] job_id 17033369
2024-03-14 07:26:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 07:26:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033369
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033369
parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17033369
parsl.executors.status_handling INFO: Allocated block ID 72
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72
2024-03-14 07:26:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:02 proc_lsst.multi:170 [INFO]  [multi] job_id 17033370
proc_lsst.multi INFO: [multi] job_id 17033370
2024-03-14 07:26:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 07:26:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033370
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033370
parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17033370
parsl.executors.status_handling INFO: Allocated block ID 73
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73
2024-03-14 07:26:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:02 proc_lsst.multi:170 [INFO]  [multi] job_id 17033371
proc_lsst.multi INFO: [multi] job_id 17033371
2024-03-14 07:26:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 07:26:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033371
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033371
parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17033371
parsl.executors.status_handling INFO: Allocated block ID 74
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74
2024-03-14 07:26:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:02 proc_lsst.multi:170 [INFO]  [multi] job_id 17033372
proc_lsst.multi INFO: [multi] job_id 17033372
2024-03-14 07:26:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 07:26:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033372
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033372
parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17033372
parsl.executors.status_handling INFO: Allocated block ID 75
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75
2024-03-14 07:26:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:02 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 07:26:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:03 proc_lsst.multi:170 [INFO]  [multi] job_id 17033373
proc_lsst.multi INFO: [multi] job_id 17033373
2024-03-14 07:26:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:03 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033373
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033373
parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17033373
parsl.executors.status_handling INFO: Allocated block ID 76
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76
2024-03-14 07:26:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:03 proc_lsst.multi:170 [INFO]  [multi] job_id 17033374
proc_lsst.multi INFO: [multi] job_id 17033374
2024-03-14 07:26:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-14 07:26:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:03 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033374
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033374
parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17033374
parsl.executors.status_handling INFO: Allocated block ID 77
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77
2024-03-14 07:26:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:03 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-14 07:26:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:04 proc_lsst.multi:170 [INFO]  [multi] job_id 17033376
proc_lsst.multi INFO: [multi] job_id 17033376
2024-03-14 07:26:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-14 07:26:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033376
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033376
parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17033376
parsl.executors.status_handling INFO: Allocated block ID 78
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78
2024-03-14 07:26:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:04 proc_lsst.multi:170 [INFO]  [multi] job_id 17033377
proc_lsst.multi INFO: [multi] job_id 17033377
2024-03-14 07:26:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-14 07:26:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033377
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033377
parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17033377
parsl.executors.status_handling INFO: Allocated block ID 79
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79
2024-03-14 07:26:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:04 proc_lsst.multi:170 [INFO]  [multi] job_id 17033378
proc_lsst.multi INFO: [multi] job_id 17033378
2024-03-14 07:26:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-14 07:26:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033378
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033378
parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17033378
parsl.executors.status_handling INFO: Allocated block ID 80
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80
2024-03-14 07:26:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:04 proc_lsst.multi:170 [INFO]  [multi] job_id 17033379
proc_lsst.multi INFO: [multi] job_id 17033379
2024-03-14 07:26:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-14 07:26:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033379
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033379
parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17033379
parsl.executors.status_handling INFO: Allocated block ID 81
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81
2024-03-14 07:26:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:04 proc_lsst.multi:170 [INFO]  [multi] job_id 17033380
proc_lsst.multi INFO: [multi] job_id 17033380
2024-03-14 07:26:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-14 07:26:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033380
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033380
parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17033380
parsl.executors.status_handling INFO: Allocated block ID 82
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82
2024-03-14 07:26:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:04 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-14 07:26:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17033381
proc_lsst.multi INFO: [multi] job_id 17033381
2024-03-14 07:26:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-14 07:26:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033381
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033381
parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17033381
parsl.executors.status_handling INFO: Allocated block ID 83
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83
2024-03-14 07:26:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17033382
proc_lsst.multi INFO: [multi] job_id 17033382
2024-03-14 07:26:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-14 07:26:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033382
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033382
parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17033382
parsl.executors.status_handling INFO: Allocated block ID 84
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84
2024-03-14 07:26:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17033383
proc_lsst.multi INFO: [multi] job_id 17033383
2024-03-14 07:26:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-14 07:26:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033383
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033383
parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17033383
parsl.executors.status_handling INFO: Allocated block ID 85
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85
2024-03-14 07:26:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17033384
proc_lsst.multi INFO: [multi] job_id 17033384
2024-03-14 07:26:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033384
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033384
parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17033384
parsl.executors.status_handling INFO: Allocated block ID 86
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86
2024-03-14 07:26:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17033385
proc_lsst.multi INFO: [multi] job_id 17033385
2024-03-14 07:26:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-14 07:26:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033385
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033385
parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17033385
parsl.executors.status_handling INFO: Allocated block ID 87
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87
2024-03-14 07:26:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-14 07:26:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:06 proc_lsst.multi:170 [INFO]  [multi] job_id 17033386
proc_lsst.multi INFO: [multi] job_id 17033386
2024-03-14 07:26:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-14 07:26:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:06 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033386
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033386
parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17033386
parsl.executors.status_handling INFO: Allocated block ID 88
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88
2024-03-14 07:26:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:06 proc_lsst.multi:170 [INFO]  [multi] job_id 17033387
proc_lsst.multi INFO: [multi] job_id 17033387
2024-03-14 07:26:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-14 07:26:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:06 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033387
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033387
parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17033387
parsl.executors.status_handling INFO: Allocated block ID 89
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89
2024-03-14 07:26:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:06 proc_lsst.multi:170 [INFO]  [multi] job_id 17033390
proc_lsst.multi INFO: [multi] job_id 17033390
2024-03-14 07:26:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-14 07:26:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:06 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033390
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033390
parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17033390
parsl.executors.status_handling INFO: Allocated block ID 90
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90
2024-03-14 07:26:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:06 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-14 07:26:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:07 proc_lsst.multi:170 [INFO]  [multi] job_id 17033391
proc_lsst.multi INFO: [multi] job_id 17033391
2024-03-14 07:26:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-14 07:26:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033391
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033391
parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17033391
parsl.executors.status_handling INFO: Allocated block ID 91
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91
2024-03-14 07:26:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:07 proc_lsst.multi:170 [INFO]  [multi] job_id 17033392
proc_lsst.multi INFO: [multi] job_id 17033392
2024-03-14 07:26:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-14 07:26:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033392
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033392
parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17033392
parsl.executors.status_handling INFO: Allocated block ID 92
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92
2024-03-14 07:26:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:07 proc_lsst.multi:170 [INFO]  [multi] job_id 17033393
proc_lsst.multi INFO: [multi] job_id 17033393
2024-03-14 07:26:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-14 07:26:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033393
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033393
parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17033393
parsl.executors.status_handling INFO: Allocated block ID 93
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93
2024-03-14 07:26:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-93
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-93
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=93 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:07 proc_lsst.multi:170 [INFO]  [multi] job_id 17033394
proc_lsst.multi INFO: [multi] job_id 17033394
2024-03-14 07:26:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 38
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38
2024-03-14 07:26:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033394
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033394
parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 17033394
parsl.executors.status_handling INFO: Allocated block ID 94
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94
2024-03-14 07:26:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-94
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-94
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 38
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38
2024-03-14 07:26:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=94 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:08 proc_lsst.multi:170 [INFO]  [multi] job_id 17033395
proc_lsst.multi INFO: [multi] job_id 17033395
2024-03-14 07:26:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 39
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39
2024-03-14 07:26:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:08 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033395
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033395
parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 17033395
parsl.executors.status_handling INFO: Allocated block ID 95
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95
2024-03-14 07:26:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-95
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-95
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 39
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=95 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:08 proc_lsst.multi:170 [INFO]  [multi] job_id 17033396
proc_lsst.multi INFO: [multi] job_id 17033396
2024-03-14 07:26:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 40
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40
2024-03-14 07:26:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:08 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033396
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033396
parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 17033396
parsl.executors.status_handling INFO: Allocated block ID 96
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96
2024-03-14 07:26:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-96
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-96
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 40
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=96 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:08 proc_lsst.multi:170 [INFO]  [multi] job_id 17033397
proc_lsst.multi INFO: [multi] job_id 17033397
2024-03-14 07:26:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 41
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41
2024-03-14 07:26:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:08 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033397
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033397
parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 17033397
parsl.executors.status_handling INFO: Allocated block ID 97
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97
2024-03-14 07:26:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-97
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-97
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:08 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 41
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41
2024-03-14 07:26:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=97 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:09 proc_lsst.multi:170 [INFO]  [multi] job_id 17033398
proc_lsst.multi INFO: [multi] job_id 17033398
2024-03-14 07:26:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 42
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42
2024-03-14 07:26:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:09 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033398
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033398
parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 17033398
parsl.executors.status_handling INFO: Allocated block ID 98
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98
2024-03-14 07:26:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-98
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-98
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 42
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=98 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:09 proc_lsst.multi:170 [INFO]  [multi] job_id 17033399
proc_lsst.multi INFO: [multi] job_id 17033399
2024-03-14 07:26:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 43
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43
2024-03-14 07:26:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:09 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033399
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033399
parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 17033399
parsl.executors.status_handling INFO: Allocated block ID 99
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99
2024-03-14 07:26:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-99
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-99
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:09 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 43
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43
2024-03-14 07:26:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=99 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:10 proc_lsst.multi:170 [INFO]  [multi] job_id 17033400
proc_lsst.multi INFO: [multi] job_id 17033400
2024-03-14 07:26:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 44
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44
2024-03-14 07:26:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:10 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033400
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033400
parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 17033400
parsl.executors.status_handling INFO: Allocated block ID 100
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100
2024-03-14 07:26:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-100
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-100
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 44
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=100 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:10 proc_lsst.multi:170 [INFO]  [multi] job_id 17033401
proc_lsst.multi INFO: [multi] job_id 17033401
2024-03-14 07:26:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 45
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45
2024-03-14 07:26:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:10 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033401
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033401
parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 17033401
parsl.executors.status_handling INFO: Allocated block ID 101
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101
2024-03-14 07:26:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-101
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-101
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 45
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=101 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:10 proc_lsst.multi:170 [INFO]  [multi] job_id 17033402
proc_lsst.multi INFO: [multi] job_id 17033402
2024-03-14 07:26:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 46
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46
2024-03-14 07:26:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:10 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033402
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033402
parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 17033402
parsl.executors.status_handling INFO: Allocated block ID 102
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102
2024-03-14 07:26:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-102
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-102
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:10 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 46
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46
2024-03-14 07:26:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=102 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:11 proc_lsst.multi:170 [INFO]  [multi] job_id 17033403
proc_lsst.multi INFO: [multi] job_id 17033403
2024-03-14 07:26:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 47
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47
2024-03-14 07:26:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033403
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033403
parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 17033403
parsl.executors.status_handling INFO: Allocated block ID 103
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103
2024-03-14 07:26:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-103
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-103
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 47
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=103 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:11 proc_lsst.multi:170 [INFO]  [multi] job_id 17033404
proc_lsst.multi INFO: [multi] job_id 17033404
2024-03-14 07:26:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 48
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48
2024-03-14 07:26:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033404
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033404
parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 17033404
parsl.executors.status_handling INFO: Allocated block ID 104
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104
2024-03-14 07:26:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-104
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-104
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 48
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=104 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:11 proc_lsst.multi:170 [INFO]  [multi] job_id 17033405
proc_lsst.multi INFO: [multi] job_id 17033405
2024-03-14 07:26:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 49
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49
2024-03-14 07:26:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033405
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033405
parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 17033405
parsl.executors.status_handling INFO: Allocated block ID 105
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105
2024-03-14 07:26:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-105
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-105
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 49
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49
2024-03-14 07:26:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=105 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:12 proc_lsst.multi:170 [INFO]  [multi] job_id 17033406
proc_lsst.multi INFO: [multi] job_id 17033406
2024-03-14 07:26:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 50
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50
2024-03-14 07:26:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:12 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033406
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033406
parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 17033406
parsl.executors.status_handling INFO: Allocated block ID 106
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106
2024-03-14 07:26:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-106
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-106
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 50
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=106 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:12 proc_lsst.multi:170 [INFO]  [multi] job_id 17033407
proc_lsst.multi INFO: [multi] job_id 17033407
2024-03-14 07:26:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 51
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51
2024-03-14 07:26:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:12 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033407
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033407
parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 17033407
parsl.executors.status_handling INFO: Allocated block ID 107
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107
2024-03-14 07:26:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-107
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-107
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 51
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=107 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 07:26:12 proc_lsst.multi:170 [INFO]  [multi] job_id 17033408
proc_lsst.multi INFO: [multi] job_id 17033408
2024-03-14 07:26:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 52
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52
2024-03-14 07:26:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:12 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033408
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033408
parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 17033408
parsl.executors.status_handling INFO: Allocated block ID 108
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108
2024-03-14 07:26:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-108
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-108
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 07:26:12 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 07:26:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 52
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52
2024-03-14 07:26:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a n3009,169.254.95.120,198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54642 --result_port=54335 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/runinfo/000/multi --block_id=108 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a680b31e069a44650742147a35c4c348 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 907895f9f62563fadf33882a6028051e
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 437
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ab604a63-f070-4bd5-977f-163d0c0c2b0a_characterizeImage_947523_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ab604a63-f070-4bd5-977f-163d0c0c2b0a_characterizeImage_947523_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/79632602-58fe-4a13-afab-84824d85f15b_isr_947523_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/79632602-58fe-4a13-afab-84824d85f15b_isr_947523_32.stderr
2024-03-14 07:26:13 proc_lsst.multi:170 [INFO]  [multi] job_id 17033409
proc_lsst.multi INFO: [multi] job_id 17033409
2024-03-14 07:26:13 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 53
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53
2024-03-14 07:26:13 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-14 07:26:13 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17033409
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17033409
parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 17033409
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:26:13 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:26:13 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:13 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:13 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:13 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:13 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf1fec07adda2e1464f0754e1e642e29 with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 11bd9bd31c12c4a38914c7028fa494b0
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 438
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/15046796-af5e-4c6c-b325-e42498e77b7d_characterizeImage_947528_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/15046796-af5e-4c6c-b325-e42498e77b7d_characterizeImage_947528_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/62ec4331-18f7-4097-894b-f712c92fdd3c_isr_947528_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/62ec4331-18f7-4097-894b-f712c92fdd3c_isr_947528_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:14 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:14 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:14 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:14 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:15 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:15 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:15 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:15 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:15 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:15 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:16 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:16 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:16 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:16 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:16 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:17 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:17 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:17 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:17 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:17 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:17 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:18 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:18 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:18 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:18 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:18 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:19 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:19 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6599a44fba57a1b4c93a5e44c834e027 with result from task 222
parsl.dataflow.dflow DEBUG: Task 222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 223 has memoization hash c057551445a9ec60f238f1ccc342dd80
parsl.dataflow.memoization INFO: Task 223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 439
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/504281c8-3610-46d5-bc41-4d4f222cc8a9_characterizeImage_947422_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/504281c8-3610-46d5-bc41-4d4f222cc8a9_characterizeImage_947422_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ca918702-58c4-4c4b-95d8-7d81f6d1c26e_isr_947422_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ca918702-58c4-4c4b-95d8-7d81f6d1c26e_isr_947422_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:19 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:19 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1092f459960700a43a9102dd6b5f094a with result from task 216
parsl.dataflow.dflow DEBUG: Task 216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 62eb9c59a9b4d531de9decd9d8ef1a1d
parsl.dataflow.memoization INFO: Task 217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 440
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/08d26375-1d48-435f-b393-4f3075e27ea3_characterizeImage_947418_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/08d26375-1d48-435f-b393-4f3075e27ea3_characterizeImage_947418_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d084e7dc-693f-41b3-a4ba-28a33ff67717_isr_947418_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d084e7dc-693f-41b3-a4ba-28a33ff67717_isr_947418_47.stderr
parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09f4e29fe35bd4e03f6b22ede7ee3acc with result from task 207
parsl.dataflow.dflow DEBUG: Task 207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 5c0e4d06c605e9934a0d9e2572c4ed8f
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 441
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/80796b49-aa4a-41ec-8529-b0f52e0ae278_characterizeImage_947524_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/80796b49-aa4a-41ec-8529-b0f52e0ae278_characterizeImage_947524_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75f61e7d-49a4-4817-8b3f-66d08d3ac3f4_isr_947524_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75f61e7d-49a4-4817-8b3f-66d08d3ac3f4_isr_947524_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:19 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:19 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:20 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12531a986830e830b2ecb477ce40106d with result from task 210
parsl.dataflow.dflow DEBUG: Task 210 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 5f4f5929578035551ef9e37c08683c07
parsl.dataflow.memoization INFO: Task 211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 442
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/63460f60-c94f-4dee-83b6-983970b33eb4_characterizeImage_947418_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/63460f60-c94f-4dee-83b6-983970b33eb4_characterizeImage_947418_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/7fecb5ab-fe04-47c7-a527-a4209efd2678_isr_947418_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/7fecb5ab-fe04-47c7-a527-a4209efd2678_isr_947418_41.stderr
parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cc2cc1c0a9e6a8a709ce60f5c7046ed with result from task 204
parsl.dataflow.dflow DEBUG: Task 204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 307e3c3e9771205b4083494c98a058ea
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 443
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/f267abe3-1593-4db0-b71d-7abe9a7c4bbc_characterizeImage_947523_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/f267abe3-1593-4db0-b71d-7abe9a7c4bbc_characterizeImage_947523_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8621e365-b272-4c54-8c82-9caaecbda208_isr_947523_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8621e365-b272-4c54-8c82-9caaecbda208_isr_947523_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:20 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:20 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52aafc66efa5d142ca75b9c3b3c8dc7d with result from task 219
parsl.dataflow.dflow DEBUG: Task 219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 220 has memoization hash aeaf0ea8c0f08296ef1942c05d0596aa
parsl.dataflow.memoization INFO: Task 220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 444
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c5e740b8-ec14-4bcd-b02a-2b4adf5dfbf1_characterizeImage_947526_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c5e740b8-ec14-4bcd-b02a-2b4adf5dfbf1_characterizeImage_947526_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/55185d5e-c44c-4350-9407-16a174b68a57_isr_947526_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/55185d5e-c44c-4350-9407-16a174b68a57_isr_947526_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:21 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:21 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:21 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02144213b0063e8cd0bb5ccf8174d8dc with result from task 201
parsl.dataflow.dflow DEBUG: Task 201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 9008014f9959d0317d53893946cc87f2
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 445
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb2a838b-c0b7-4f89-970d-d14a93308481_characterizeImage_947422_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb2a838b-c0b7-4f89-970d-d14a93308481_characterizeImage_947422_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f5cdf599-3dd4-4dbb-8a4b-0e98118fd6b2_isr_947422_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f5cdf599-3dd4-4dbb-8a4b-0e98118fd6b2_isr_947422_59.stderr
parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4afd265e5a0681d6ee78e9654d528ec4 with result from task 213
parsl.dataflow.dflow DEBUG: Task 213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 6bbf48affe83937415e9a9c06006b669
parsl.dataflow.memoization INFO: Task 214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 446
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/31c1ff00-0114-4524-946a-f37fadad90d4_characterizeImage_947418_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/31c1ff00-0114-4524-946a-f37fadad90d4_characterizeImage_947418_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b3713cf3-72ed-4885-8da1-2d66f53d608a_isr_947418_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b3713cf3-72ed-4885-8da1-2d66f53d608a_isr_947418_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:22 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:22 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92d0c22ea88b3b617621a91709265c71 with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash b5ee63d883835cdc888c3788fbfb0a8c
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 447
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1a321fd6-822d-4596-af2c-ec61d1c44cae_characterizeImage_947524_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1a321fd6-822d-4596-af2c-ec61d1c44cae_characterizeImage_947524_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f714ed3b-1da7-4bba-864e-03508fe9e2ea_isr_947524_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f714ed3b-1da7-4bba-864e-03508fe9e2ea_isr_947524_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:22 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:22 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:22 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:23 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:23 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:23 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:23 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97f2d1afc3e505fb1755f624732f08a7 with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 53d61f5f65d75c7e6f41998af69234ce
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 448
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb4a7d5c-8f33-409e-b14a-2b2b1bddfcbf_characterizeImage_947422_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb4a7d5c-8f33-409e-b14a-2b2b1bddfcbf_characterizeImage_947422_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/436e0914-6ce5-4eec-9240-1e61a127c9a7_isr_947422_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/436e0914-6ce5-4eec-9240-1e61a127c9a7_isr_947422_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:23 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6cf3b4862d2a989e0dcd3b5dfca5ce5 with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash d051c6579ae10e8048fd3b06191d8123
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 449
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2709140e-7397-4b99-a802-6f4c132f6a7b_characterizeImage_947526_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2709140e-7397-4b99-a802-6f4c132f6a7b_characterizeImage_947526_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/86490b56-258d-41dc-8d44-68a2777b1207_isr_947526_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/86490b56-258d-41dc-8d44-68a2777b1207_isr_947526_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:23 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:24 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:24 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:24 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:24 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:24 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65fb7f734d7b3b0f0dcc622f1e39144c with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 95a12943d400bd338eb34d2e3438e0d0
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 450
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/78612383-bc15-4f35-9c57-b580d53b6b0a_characterizeImage_947420_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/78612383-bc15-4f35-9c57-b580d53b6b0a_characterizeImage_947420_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54874989-ff39-4d0c-939f-86b577b9d889_isr_947420_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54874989-ff39-4d0c-939f-86b577b9d889_isr_947420_27.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f510aadef85a68f587353cb51e31874d with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 20b61fb74507be75703cd1c00572fedb
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 451
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7ffe30d6-f557-4a05-9f64-ecdca4cf38e7_characterizeImage_947528_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7ffe30d6-f557-4a05-9f64-ecdca4cf38e7_characterizeImage_947528_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dd084417-5f1c-4ff9-bb09-badcda5f8116_isr_947528_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dd084417-5f1c-4ff9-bb09-badcda5f8116_isr_947528_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:25 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:25 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8599f0fc209f949ea2f7d4e18a222dc7 with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 9702670b5282678251a3ef532036e777
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 452
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9820409f-45aa-4154-a3a3-0fa61ced5422_characterizeImage_947420_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9820409f-45aa-4154-a3a3-0fa61ced5422_characterizeImage_947420_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/424d4475-9099-4c20-80c7-f95958d31f72_isr_947420_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/424d4475-9099-4c20-80c7-f95958d31f72_isr_947420_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:25 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed63ef2ed102c4604725ca232066d5c7 with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 03179fa9c2a6477dc76c111cb6de8379
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 453
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/43c6985a-cf38-452d-a2b2-08d4646e4acd_characterizeImage_947420_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/43c6985a-cf38-452d-a2b2-08d4646e4acd_characterizeImage_947420_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2647de5a-5d12-43d2-b9d2-e973f021e183_isr_947420_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2647de5a-5d12-43d2-b9d2-e973f021e183_isr_947420_8.stderr
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e153b32267663cf0b85d7bcad21e7487 with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 89f1973f4e5820a84f0488a40098584a
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 454
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/dc228e9b-39ea-47af-9780-f5050875104a_characterizeImage_947523_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/dc228e9b-39ea-47af-9780-f5050875104a_characterizeImage_947523_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8a8cb7e3-3285-4d3a-a2ba-99a1534a361c_isr_947523_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8a8cb7e3-3285-4d3a-a2ba-99a1534a361c_isr_947523_13.stderr
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c9009d078f68c517a2b467204336356 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash cf1507d917f6debeda6ef688866fe18d
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 455
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cb4f827f-9be2-480e-87e2-d6dabe86bc8c_characterizeImage_947418_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cb4f827f-9be2-480e-87e2-d6dabe86bc8c_characterizeImage_947418_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2af53ff0-f16c-46dd-88a8-8252892d092c_isr_947418_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2af53ff0-f16c-46dd-88a8-8252892d092c_isr_947418_21.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 879602bfe3bb6938fac51e48fc48e3eb with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash f52133c384507fdb19e7b724ca405b80
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 456
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/551690df-ce1f-459c-a693-627391d96a30_characterizeImage_947524_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/551690df-ce1f-459c-a693-627391d96a30_characterizeImage_947524_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/efb96d0e-54cc-454e-a3f1-6e27b6621648_isr_947524_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/efb96d0e-54cc-454e-a3f1-6e27b6621648_isr_947524_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:26 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6ca5171a391c689bf2db30f7f8c2d77 with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 370264e1c86d44e8cad5e4cd5af89c79
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 457
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6ef38294-9cbd-4387-b71c-0582e9a6f704_characterizeImage_947422_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6ef38294-9cbd-4387-b71c-0582e9a6f704_characterizeImage_947422_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/64d7f367-b32c-4c18-9463-77dd7da53f63_isr_947422_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/64d7f367-b32c-4c18-9463-77dd7da53f63_isr_947422_27.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58a29cba9d1cae8f2990969b304f229e with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 2506e7e0dbea666f339cc9ceec680fb7
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 458
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f988318d-eacf-4d02-a10b-25b88f52a56f_characterizeImage_947422_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f988318d-eacf-4d02-a10b-25b88f52a56f_characterizeImage_947422_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7b203241-bc8e-425d-93de-b3af02d8ed22_isr_947422_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7b203241-bc8e-425d-93de-b3af02d8ed22_isr_947422_40.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f0ff4cfa702b223a27d0128195ca740 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 20ce315cfae902bb6dd078db69cc62fd
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 459
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/af825a7f-a7be-4e6b-b394-514fa0162b28_characterizeImage_947523_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/af825a7f-a7be-4e6b-b394-514fa0162b28_characterizeImage_947523_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f91398c4-7511-4251-a6d4-08a581530dbc_isr_947523_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f91398c4-7511-4251-a6d4-08a581530dbc_isr_947523_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:26 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 088f8ffda1b2936c7f4d48f73cfcb020 with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash dc10d105e2e8114a6b3e77728d88fa1b
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 460
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b51cd98-1124-4008-887f-5373be0c9ad4_characterizeImage_947526_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b51cd98-1124-4008-887f-5373be0c9ad4_characterizeImage_947526_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a8c3f37c-cec8-408d-b35a-aa9f42eb7c59_isr_947526_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a8c3f37c-cec8-408d-b35a-aa9f42eb7c59_isr_947526_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:26 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:26 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5439168b6847fc3f4db1b36a87b09470 with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash f54a8e1c3360a8a743811186ddb02ce4
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 461
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d98895c3-5024-4011-b618-b255cf25ce4d_characterizeImage_947524_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d98895c3-5024-4011-b618-b255cf25ce4d_characterizeImage_947524_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2566c465-ae72-4e64-9626-3ce218d0ed86_isr_947524_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2566c465-ae72-4e64-9626-3ce218d0ed86_isr_947524_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:27 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:27 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd40a6b1cd7c2a3ccfe8473bbfe673b with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash ffab501d781a9e638ae00420bc43b192
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 462
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9d68d909-5f05-490f-8403-50028819c5aa_characterizeImage_947420_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9d68d909-5f05-490f-8403-50028819c5aa_characterizeImage_947420_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54438edd-a48b-41d7-a80c-9e665e40d219_isr_947420_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/54438edd-a48b-41d7-a80c-9e665e40d219_isr_947420_7.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc3aab7c8de6b2e3409ddcb48131a059 with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash f6954da9d490a1be32eea10181b8ea0d
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 463
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/685a7bbe-0e9f-4da2-b106-1acc0c445408_characterizeImage_947524_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/685a7bbe-0e9f-4da2-b106-1acc0c445408_characterizeImage_947524_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/106ecfbf-126f-4712-bed2-4457add4682f_isr_947524_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/106ecfbf-126f-4712-bed2-4457add4682f_isr_947524_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:27 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:28 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:28 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6fea76f493b609470e8f2f54b2e72f82 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 64a57e1c93a758c68f8ebd5f13abf6ae
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 464
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a71beaf-22de-4260-903c-9fedf146a209_characterizeImage_947524_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a71beaf-22de-4260-903c-9fedf146a209_characterizeImage_947524_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2edb5942-9fd3-4102-9cb3-401e260cb7f4_isr_947524_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2edb5942-9fd3-4102-9cb3-401e260cb7f4_isr_947524_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:29 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:29 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65f58f75544c3ee8067760562e1c2c09 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 3905bc78c6323720653dd62c4ee08ee0
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 465
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/42830f79-c4c7-4845-b53c-d74e2bed84c9_characterizeImage_947524_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/42830f79-c4c7-4845-b53c-d74e2bed84c9_characterizeImage_947524_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a9dc563-c252-4759-8c01-923a6675a660_isr_947524_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a9dc563-c252-4759-8c01-923a6675a660_isr_947524_57.stderr
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc5fc4558e85c206d571729c75bb227b with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 795cb9a82decc6d6d76f9ad7a8566f2c
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 466
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4b2e36e6-3f51-4499-9aa0-43b7de685f57_characterizeImage_947418_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4b2e36e6-3f51-4499-9aa0-43b7de685f57_characterizeImage_947418_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/ffec0d9e-4c97-4604-89a8-e3702956512f_isr_947418_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/ffec0d9e-4c97-4604-89a8-e3702956512f_isr_947418_39.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 477729f1754dc8155c0f3780712d8d91 with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 4bbbbf1a15d48d07b31eb9e45a4cdbe9
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 467
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/e2c156d6-acbf-484c-a19e-1942d7d0ada2_characterizeImage_947524_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/e2c156d6-acbf-484c-a19e-1942d7d0ada2_characterizeImage_947524_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ca052227-1f08-4f19-a9e8-d022844d8894_isr_947524_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ca052227-1f08-4f19-a9e8-d022844d8894_isr_947524_23.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59c49320808107f4ef7fb2ebdb4d557a with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 2acffa80cb895c3c71708959d2b4763d
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 468
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/20004bf9-e33f-4fc5-aaa0-c600df3e0e0d_characterizeImage_947523_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/20004bf9-e33f-4fc5-aaa0-c600df3e0e0d_characterizeImage_947523_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2d2f199c-373b-46f5-bde1-87ad00150a36_isr_947523_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2d2f199c-373b-46f5-bde1-87ad00150a36_isr_947523_57.stderr
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc8175d2f63a462db8989b4152a7878 with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash bcf936070c007b64857775d4a6c48396
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 469
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/09245fe7-31cf-48d2-a0d5-62490e1336a2_characterizeImage_947420_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/09245fe7-31cf-48d2-a0d5-62490e1336a2_characterizeImage_947420_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/acde34d5-141f-4576-b8d7-bc5a8c67357b_isr_947420_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/acde34d5-141f-4576-b8d7-bc5a8c67357b_isr_947420_13.stderr
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04b384bfd5067004c447ce97f7e5c108 with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash f969fda4f595861671a8d6d6cecb399c
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 470
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/337c0eff-5310-4e6a-aead-449b628c4b1e_characterizeImage_947528_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/337c0eff-5310-4e6a-aead-449b628c4b1e_characterizeImage_947528_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/60d16572-8a1c-434f-a11d-87c159dfe6fa_isr_947528_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/60d16572-8a1c-434f-a11d-87c159dfe6fa_isr_947528_19.stderr
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7f8a5d2a5c2392bbd466007b68f0a2e with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash ec7722f3643a9303e4b77faba46de362
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e954e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 471
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/21f9be78-c3fb-4e48-9e11-ef3de58a00ca_characterizeImage_947526_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/21f9be78-c3fb-4e48-9e11-ef3de58a00ca_characterizeImage_947526_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/680cb945-a656-4ed6-b830-cc6cbac4eb3d_isr_947526_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/680cb945-a656-4ed6-b830-cc6cbac4eb3d_isr_947526_13.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 705e53d1704bf3fa08e2a1d3299e9a42 with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 4d3396e49dd6a3f2e7e1244cbce94130
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 472
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5eaf682c-b1b1-4a67-8f8e-17acb3222e15_characterizeImage_947523_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5eaf682c-b1b1-4a67-8f8e-17acb3222e15_characterizeImage_947523_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b6a5f6d7-5b56-4ce5-8d04-fdd48f6e7a0f_isr_947523_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b6a5f6d7-5b56-4ce5-8d04-fdd48f6e7a0f_isr_947523_36.stderr
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef02268a7f9cf90a487582d767d973d2 with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash bcff50dc1e73932270c5eef16c9e1a73
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 473
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5a32c7a6-1408-4fe6-b546-657eae51ef23_characterizeImage_947528_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5a32c7a6-1408-4fe6-b546-657eae51ef23_characterizeImage_947528_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7e5cc74c-101d-4f6f-b453-ebf6401bc599_isr_947528_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7e5cc74c-101d-4f6f-b453-ebf6401bc599_isr_947528_26.stderr
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dab2031f935fdcb5da9790e6d77692aa with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash cde30515c2f5f85a6092c957fc8b1172
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 474
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0fb71db5-9272-4dff-a8d0-c0d8de04726e_characterizeImage_947422_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0fb71db5-9272-4dff-a8d0-c0d8de04726e_characterizeImage_947422_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/eea00512-d9dd-4da4-bb0c-8c24620753cc_isr_947422_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/eea00512-d9dd-4da4-bb0c-8c24620753cc_isr_947422_5.stderr
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9501c7fada462cdec09b29306952e4c0 with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash bfe6e8b90c831e4e6a835de578727254
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 475
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/4eddb58d-0f24-4009-af06-56641fd1af49_characterizeImage_947526_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/4eddb58d-0f24-4009-af06-56641fd1af49_characterizeImage_947526_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/75da4b06-9599-4f47-8eef-9115c02d470d_isr_947526_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/75da4b06-9599-4f47-8eef-9115c02d470d_isr_947526_52.stderr
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 447c340cbba14e5c357ecf29bdf79c8d with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash cbfe4eee516760fccf42f47f5d75a0b2
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 476
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/764d1136-bb72-42df-bc7c-66de51da84b1_characterizeImage_947524_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/764d1136-bb72-42df-bc7c-66de51da84b1_characterizeImage_947524_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ee80558e-a70b-4e42-9277-bfe16a45eed3_isr_947524_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ee80558e-a70b-4e42-9277-bfe16a45eed3_isr_947524_36.stderr
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3a140c5a8c82950372d2f0fdcd41bf74 with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 31d26feb67260add5faca529f6684723
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 477
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/e43dbd4c-e3ad-45b2-b3a8-559e0cf4be00_characterizeImage_947528_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/e43dbd4c-e3ad-45b2-b3a8-559e0cf4be00_characterizeImage_947528_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1d2c75d2-b93f-44b4-9843-e8c536eb1c15_isr_947528_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1d2c75d2-b93f-44b4-9843-e8c536eb1c15_isr_947528_44.stderr
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 036f61cbfdd1b5fd95e37ed3df1390ce with result from task 84
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:29 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 64eb349dd9c93a22ef7083e2f9c2844e
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 478
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ba42188d-afdb-44d6-9a21-5a59fd186fd1_characterizeImage_947528_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ba42188d-afdb-44d6-9a21-5a59fd186fd1_characterizeImage_947528_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4b183dc3-bb24-4d99-8cab-077a420134a3_isr_947528_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4b183dc3-bb24-4d99-8cab-077a420134a3_isr_947528_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:30 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:30 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:30 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:30 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:30 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:31 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:31 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:31 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:32 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:32 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:32 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:32 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:32 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:32 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:33 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b61eab59287ea052a7a6f7e6a41c085a with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 7d9fff4c642f960bb13e0e697431172e
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 479
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/da9c948a-73f4-4b52-8201-b985b0819e3f_characterizeImage_947524_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/da9c948a-73f4-4b52-8201-b985b0819e3f_characterizeImage_947524_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/95365e2c-8aa5-4f35-9f62-e42306772a12_isr_947524_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/95365e2c-8aa5-4f35-9f62-e42306772a12_isr_947524_14.stderr
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b646098fbb2c2badcc27a69e6f2ba39c with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 0fdfe2b01c703530968a7bf09ede4e68
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 480
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/74acadc3-9f20-4a58-b812-3361a2fa46fa_characterizeImage_947418_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/74acadc3-9f20-4a58-b812-3361a2fa46fa_characterizeImage_947418_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/24ce32cd-7873-4754-892d-5432ceba0974_isr_947418_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/24ce32cd-7873-4754-892d-5432ceba0974_isr_947418_27.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c9edd49eb171a9f98853ebe592518cf with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 64f129cca316e8c8b77408b741cc047d
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 481
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/47ca7f3c-ad65-4a75-a8eb-625c16822815_characterizeImage_947420_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/47ca7f3c-ad65-4a75-a8eb-625c16822815_characterizeImage_947420_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/374fcd2a-b964-476c-8fa0-d4c4d98968a9_isr_947420_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/374fcd2a-b964-476c-8fa0-d4c4d98968a9_isr_947420_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:33 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:33 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0afda3c81ac8620cf4847da0ac722f43 with result from task 915
parsl.dataflow.dflow DEBUG: Task 915 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 916 has memoization hash e497af5e9225edfd599bb8e4a2103679
parsl.dataflow.memoization INFO: Task 916 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 482
parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8d9dad21-1939-4051-830f-2f553f67640c_characterizeImage_947422_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8d9dad21-1939-4051-830f-2f553f67640c_characterizeImage_947422_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5913627c-b237-46d2-8e00-97694b6e7ff1_isr_947422_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5913627c-b237-46d2-8e00-97694b6e7ff1_isr_947422_57.stderr
parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99985cafa158e0d7c350684a9365dd87 with result from task 882
parsl.dataflow.dflow DEBUG: Task 882 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 0ae9581883d6120b7f7c1a12b73d8c9d
parsl.dataflow.memoization INFO: Task 883 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 483
parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/deda3783-4c05-40a1-afad-9060380ee177_characterizeImage_947528_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/deda3783-4c05-40a1-afad-9060380ee177_characterizeImage_947528_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/a2a9cc77-8ed8-4e17-99f7-653d6cba04a6_isr_947528_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/a2a9cc77-8ed8-4e17-99f7-653d6cba04a6_isr_947528_18.stderr
parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6542a3c74f2431abf97905de3320b3e5 with result from task 879
parsl.dataflow.dflow DEBUG: Task 879 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 880 has memoization hash fe6dd44c3080e22e8b2262ee534d1933
parsl.dataflow.memoization INFO: Task 880 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 484
parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b239acb1-3828-4a4a-83c1-59b662f7e91b_characterizeImage_947524_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b239acb1-3828-4a4a-83c1-59b662f7e91b_characterizeImage_947524_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/7a714680-a13e-4d8c-8f9e-6c08174fa906_isr_947524_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/7a714680-a13e-4d8c-8f9e-6c08174fa906_isr_947524_18.stderr
parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d59f67dbda6c74c7b67c4a96cf3c3cf2 with result from task 885
parsl.dataflow.dflow DEBUG: Task 885 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 0b71b0d8ddcc53d5f06b5a9eac577ed7
parsl.dataflow.memoization INFO: Task 886 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 485
parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/89df2ad1-0c3a-4ca5-b25c-2b8a0f4c4da6_characterizeImage_947422_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/89df2ad1-0c3a-4ca5-b25c-2b8a0f4c4da6_characterizeImage_947422_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6bf623b7-72cb-4dd4-b5f4-cd8f5c75e6cf_isr_947422_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6bf623b7-72cb-4dd4-b5f4-cd8f5c75e6cf_isr_947422_17.stderr
parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8744365a5dfb8a23dc46a7f2450874d2 with result from task 465
parsl.dataflow.dflow DEBUG: Task 465 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 8a0a05a34ba63c3de245c65cc9e2c270
parsl.dataflow.memoization INFO: Task 466 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 486
parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2cdfa2bf-e496-4c30-906d-3569c9bac572_characterizeImage_947523_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2cdfa2bf-e496-4c30-906d-3569c9bac572_characterizeImage_947523_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/a24136e6-b29a-46aa-900d-158bdec834ff_isr_947523_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/a24136e6-b29a-46aa-900d-158bdec834ff_isr_947523_45.stderr
parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7a1bbd096de41295c4c209314fccb4f with result from task 588
parsl.dataflow.dflow DEBUG: Task 588 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 589 has memoization hash cf33cf693bde9c1deb92d4b851fddd0e
parsl.dataflow.memoization INFO: Task 589 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 487
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/524fab00-7aa7-4f0b-ae56-296bdc09bcff_characterizeImage_947422_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/524fab00-7aa7-4f0b-ae56-296bdc09bcff_characterizeImage_947422_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/99cb83ab-e2eb-417f-bd5d-6641b94e015c_isr_947422_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/99cb83ab-e2eb-417f-bd5d-6641b94e015c_isr_947422_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2fbdacf074a2e022d40646399688353 with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 52201ba983121e6344066f61a8d31e6a
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 488
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/8fd17842-7b45-45de-97e7-99441a69c70c_characterizeImage_947420_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/8fd17842-7b45-45de-97e7-99441a69c70c_characterizeImage_947420_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/6118c3d2-f2f8-425a-895d-fb06a5816e43_isr_947420_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/6118c3d2-f2f8-425a-895d-fb06a5816e43_isr_947420_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afbda6e30937b15f34c41c3181475dea with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 0d6c7c820b5d66aaff86505dd7fbb00e
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 489
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6fe130c-9256-4300-9cd4-b274e26530fa_characterizeImage_947523_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6fe130c-9256-4300-9cd4-b274e26530fa_characterizeImage_947523_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/e5ce27d4-a452-4b74-94a6-cd18199b2808_isr_947523_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/e5ce27d4-a452-4b74-94a6-cd18199b2808_isr_947523_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:34 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e08f00a04f6dc86c03d1c6ca11b453f2 with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 62e924eb7c241e5a30f062d8b4feb5d0
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 490
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a28b34d5-d261-4aeb-909b-e4201fdb9a47_characterizeImage_947418_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a28b34d5-d261-4aeb-909b-e4201fdb9a47_characterizeImage_947418_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/55d97697-dd9a-47ea-8649-d676347ea4a5_isr_947418_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/55d97697-dd9a-47ea-8649-d676347ea4a5_isr_947418_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38dd61d1db923aa18e71130827a76c1d with result from task 912
parsl.dataflow.dflow DEBUG: Task 912 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 917960c9fd1c9812da0e15fa93098f1d
parsl.dataflow.memoization INFO: Task 913 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 491
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/537fe35e-2f10-4983-9f12-b17a199d9ad6_characterizeImage_947418_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/537fe35e-2f10-4983-9f12-b17a199d9ad6_characterizeImage_947418_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/894d900c-79f7-4bd7-b9a4-62392d3bf7b7_isr_947418_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/894d900c-79f7-4bd7-b9a4-62392d3bf7b7_isr_947418_54.stderr
parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9df982db57fd977e40c131225c45241c with result from task 1065
parsl.dataflow.dflow DEBUG: Task 1065 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash f5d953832bf9abfe2c2c815eb7a9383d
parsl.dataflow.memoization INFO: Task 1066 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 492
parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6b7446b5-8529-4fea-b610-f332e19e4195_characterizeImage_947524_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6b7446b5-8529-4fea-b610-f332e19e4195_characterizeImage_947524_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ff61384-0bbd-4c56-b95f-6b89c97a7747_isr_947524_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ff61384-0bbd-4c56-b95f-6b89c97a7747_isr_947524_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:35 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:36 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 434
parsl.dataflow.strategy DEBUG: Executor multi has 434 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 434
parsl.dataflow.strategy DEBUG: Executor multi has 434 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dad0c02112a0b6cb2c31b6bfc82a381b with result from task 966
parsl.dataflow.dflow DEBUG: Task 966 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 967 has memoization hash d3f217129036e2b198a382ce8fc48c3b
parsl.dataflow.memoization INFO: Task 967 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 493
parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/08142465-1f37-4e4b-ae1c-5710b37b6f8f_characterizeImage_947524_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/08142465-1f37-4e4b-ae1c-5710b37b6f8f_characterizeImage_947524_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4c39db58-65ad-4ce7-8527-37b007391047_isr_947524_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4c39db58-65ad-4ce7-8527-37b007391047_isr_947524_40.stderr
parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9e7201be891a5f86e667410f459a45b with result from task 237
parsl.dataflow.dflow DEBUG: Task 237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash fa4e6fda98cdbf97c82cbcee5d8f4308
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 494
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c0ef73ae-6759-4a8b-b557-d0c0281970b2_isr_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c0ef73ae-6759-4a8b-b557-d0c0281970b2_isr_947420_2.stderr
parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 766caba1caf75da452038f16823489ab with result from task 267
parsl.dataflow.dflow DEBUG: Task 267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 491c2c2ee149832357064e3d3ed2d00a
parsl.dataflow.memoization INFO: Task 268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 495
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/efaf004a-6057-417f-8209-91294f59ae47_characterizeImage_947528_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/efaf004a-6057-417f-8209-91294f59ae47_characterizeImage_947528_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/28f32647-caa3-47fc-8a6f-97ae950bc62c_isr_947528_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/28f32647-caa3-47fc-8a6f-97ae950bc62c_isr_947528_31.stderr
parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76eda758a0c1a2b146e68e3eec307d7a with result from task 261
parsl.dataflow.dflow DEBUG: Task 261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 262 has memoization hash f8f634dea45932bdf5a08ac628236ea9
parsl.dataflow.memoization INFO: Task 262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 496
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/6590c257-5d37-440c-9f48-14cf6784d949_characterizeImage_947526_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/6590c257-5d37-440c-9f48-14cf6784d949_characterizeImage_947526_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c84583a9-0200-4b1b-93af-424d56042ce4_isr_947526_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c84583a9-0200-4b1b-93af-424d56042ce4_isr_947526_57.stderr
parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4bce17763315288a6fec69ca9acb745 with result from task 1068
parsl.dataflow.dflow DEBUG: Task 1068 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash a77119c26b3c069b2aaacf01ee34b912
parsl.dataflow.memoization INFO: Task 1069 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 497
parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/5a63493c-1b05-4c53-b781-975d63d04235_characterizeImage_947420_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/5a63493c-1b05-4c53-b781-975d63d04235_characterizeImage_947420_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/332fe5e0-6d63-4c4d-a34a-c43f05991797_isr_947420_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/332fe5e0-6d63-4c4d-a34a-c43f05991797_isr_947420_49.stderr
parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aff8cee467ef3d4af4aac693d5434241 with result from task 255
parsl.dataflow.dflow DEBUG: Task 255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 256 has memoization hash d9399a1ab5a7031a52ab3e20b556aa62
parsl.dataflow.memoization INFO: Task 256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 498
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f0a42a5b-7f27-47fd-8e96-12654c01ba68_characterizeImage_947528_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f0a42a5b-7f27-47fd-8e96-12654c01ba68_characterizeImage_947528_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/42d071ee-2b6d-4817-a039-9cb236c964da_isr_947528_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/42d071ee-2b6d-4817-a039-9cb236c964da_isr_947528_8.stderr
parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4b1e6e0c2e3510a36b48dd742e28849 with result from task 249
parsl.dataflow.dflow DEBUG: Task 249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 574d308631b722e7897bbee107822866
parsl.dataflow.memoization INFO: Task 250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 499
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1fab9ce6-d5ab-41f6-b036-c902fb7774a9_characterizeImage_947528_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1fab9ce6-d5ab-41f6-b036-c902fb7774a9_characterizeImage_947528_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/66de8d43-aef4-4c29-aa81-4d577ba8242e_isr_947528_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/66de8d43-aef4-4c29-aa81-4d577ba8242e_isr_947528_16.stderr
parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee98e535ce7ba1975341a7e3f133d0b0 with result from task 258
parsl.dataflow.dflow DEBUG: Task 258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 9604a776e533cf26dce9b127ff0dfc22
parsl.dataflow.memoization INFO: Task 259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 500
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/61d102c0-2d53-4d7e-b401-94c550e6b1c5_characterizeImage_947528_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/61d102c0-2d53-4d7e-b401-94c550e6b1c5_characterizeImage_947528_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b36ed276-870f-4838-8eaa-03b7600cc3c7_isr_947528_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b36ed276-870f-4838-8eaa-03b7600cc3c7_isr_947528_45.stderr
parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52434a50e159a1b9c29e5840a1737992 with result from task 225
parsl.dataflow.dflow DEBUG: Task 225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash c0d2cd095fbb412acd645460259402fd
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 501
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4e87ca2b-79d6-4eac-b64b-a532dcf04f8d_isr_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4e87ca2b-79d6-4eac-b64b-a532dcf04f8d_isr_947523_2.stderr
parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa148597e3bc233f1698475a7d5e0e0c with result from task 282
parsl.dataflow.dflow DEBUG: Task 282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 283 has memoization hash b570ffaa9231a348da7c0853f33d2adf
parsl.dataflow.memoization INFO: Task 283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 502
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b0429d5-d0d2-4681-b4b6-607b6e889b58_characterizeImage_947418_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b0429d5-d0d2-4681-b4b6-607b6e889b58_characterizeImage_947418_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e8bafed2-bfe4-45e6-aeb9-7be2643a1765_isr_947418_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e8bafed2-bfe4-45e6-aeb9-7be2643a1765_isr_947418_51.stderr
parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73f3a6e3db26a83edff7115c85775bc4 with result from task 294
parsl.dataflow.dflow DEBUG: Task 294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 794b798d2626f76cc612ea75f1d06020
parsl.dataflow.memoization INFO: Task 295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 503
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/60900f45-0e04-4e53-910a-916fde56226e_characterizeImage_947523_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/60900f45-0e04-4e53-910a-916fde56226e_characterizeImage_947523_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/66811516-e824-4465-aca2-9fb7cad5c3b9_isr_947523_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/66811516-e824-4465-aca2-9fb7cad5c3b9_isr_947523_19.stderr
parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92a7cece00aac9a5ad085c0dbb0491cd with result from task 252
parsl.dataflow.dflow DEBUG: Task 252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 255ccbc055311f76526de4acebb53c74
parsl.dataflow.memoization INFO: Task 253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 504
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/22e5da8f-b83d-43da-ba9d-d17c1547c7bf_characterizeImage_947418_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/22e5da8f-b83d-43da-ba9d-d17c1547c7bf_characterizeImage_947418_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8a342e0e-0b63-412a-8f1e-cbd940529c28_isr_947418_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8a342e0e-0b63-412a-8f1e-cbd940529c28_isr_947418_16.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 434
parsl.dataflow.strategy DEBUG: Executor multi has 434 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc6344ddc91dfd4211ffb9ebb6a6c78d with result from task 264
parsl.dataflow.dflow DEBUG: Task 264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 274fd0c6fb5bee99c13c9009379ec76e
parsl.dataflow.memoization INFO: Task 265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 505
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/777be1ce-f60a-4288-99e8-718cf90a4db2_characterizeImage_947526_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/777be1ce-f60a-4288-99e8-718cf90a4db2_characterizeImage_947526_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9cc439b7-233e-4410-9342-92d50676c567_isr_947526_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9cc439b7-233e-4410-9342-92d50676c567_isr_947526_42.stderr
parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d639087b200327dc92a0fef77637a9f with result from task 273
parsl.dataflow.dflow DEBUG: Task 273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 8b31860103a049d5af619ba1e0e5f365
parsl.dataflow.memoization INFO: Task 274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f007c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 506
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d86f5e71-54e6-4069-ba5d-23c516302c95_characterizeImage_947524_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d86f5e71-54e6-4069-ba5d-23c516302c95_characterizeImage_947524_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/3da3af2f-f3f4-49c6-94cd-4b86764a97f0_isr_947524_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/3da3af2f-f3f4-49c6-94cd-4b86764a97f0_isr_947524_53.stderr
parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d3b39d5403c2968085cb380e467f59b with result from task 276
parsl.dataflow.dflow DEBUG: Task 276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 0f1680e68c53582dd573b0ce1ab35fbc
parsl.dataflow.memoization INFO: Task 277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 507
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3ed52454-78ed-4675-975e-05c6cf3c362f_characterizeImage_947523_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3ed52454-78ed-4675-975e-05c6cf3c362f_characterizeImage_947523_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cad49c99-d089-4cf0-9fee-b85261707ee0_isr_947523_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cad49c99-d089-4cf0-9fee-b85261707ee0_isr_947523_47.stderr
parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46143bd2aca114be53d56d154c8cb9c8 with result from task 291
parsl.dataflow.dflow DEBUG: Task 291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 292 has memoization hash b04601b9a4f19b29b7d76a80e1019575
parsl.dataflow.memoization INFO: Task 292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 508
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7071bda8-6073-47aa-9d52-7bd022c68cbe_characterizeImage_947524_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7071bda8-6073-47aa-9d52-7bd022c68cbe_characterizeImage_947524_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/96f56684-2f90-4643-9574-71f6f1eab4b3_isr_947524_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/96f56684-2f90-4643-9574-71f6f1eab4b3_isr_947524_8.stderr
parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa10a5b269e51d3d79eae7ae7603cde8 with result from task 351
parsl.dataflow.dflow DEBUG: Task 351 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 4cfec2f9edbbcea86f056e813319bd42
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 509
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/372bc865-ea4b-40de-b127-3f7c06a3e12b_isr_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/372bc865-ea4b-40de-b127-3f7c06a3e12b_isr_947422_2.stderr
parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 05877509337712ba848e115404b5404d with result from task 285
parsl.dataflow.dflow DEBUG: Task 285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 5ef87dfc8c5b4dc4d4de9b9a857bef11
parsl.dataflow.memoization INFO: Task 286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 510
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e443c860-8a14-49ff-a2f3-912fa54126bc_characterizeImage_947418_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e443c860-8a14-49ff-a2f3-912fa54126bc_characterizeImage_947418_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f9cf9043-618c-4099-8a4d-c6d870d9c1cc_isr_947418_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f9cf9043-618c-4099-8a4d-c6d870d9c1cc_isr_947418_10.stderr
parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51efb2ef95aa6f6912c0e7f91760e87e with result from task 279
parsl.dataflow.dflow DEBUG: Task 279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 0e4fb67619d77f85c20af1528fa393ca
parsl.dataflow.memoization INFO: Task 280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 511
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0c958d5b-9f5e-44de-a1cd-06850071ebcc_characterizeImage_947420_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0c958d5b-9f5e-44de-a1cd-06850071ebcc_characterizeImage_947420_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bbd78b0-b53e-4bf8-9b3d-aeb6990066d7_isr_947420_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bbd78b0-b53e-4bf8-9b3d-aeb6990066d7_isr_947420_25.stderr
parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d6c05285e1eea24c6d10ce13a7cb4ef with result from task 288
parsl.dataflow.dflow DEBUG: Task 288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 72c4189881b5d6540b267e721bf9e8db
parsl.dataflow.memoization INFO: Task 289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 512
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be4ec842-adce-4bfa-8a04-b4dee174d0dc_characterizeImage_947420_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be4ec842-adce-4bfa-8a04-b4dee174d0dc_characterizeImage_947420_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8cb87310-6730-42bb-8977-e0bc8e9f70dc_isr_947420_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8cb87310-6730-42bb-8977-e0bc8e9f70dc_isr_947420_43.stderr
parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 290b17c6f1b73cc315c23149a4c85e2d with result from task 306
parsl.dataflow.dflow DEBUG: Task 306 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 542b037d1e04a8ce828b1263d1e0dea6
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 513
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/67d57fc7-a298-45aa-9f70-9970f36f1530_characterizeImage_947523_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/67d57fc7-a298-45aa-9f70-9970f36f1530_characterizeImage_947523_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9e409e6a-1549-44c7-9c7f-397a83960109_isr_947523_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9e409e6a-1549-44c7-9c7f-397a83960109_isr_947523_22.stderr
parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f94f9e218d67fc22d57d383efdfa164 with result from task 303
parsl.dataflow.dflow DEBUG: Task 303 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 304 has memoization hash f1f617c955212c09a89ff7495844399a
parsl.dataflow.memoization INFO: Task 304 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 514
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3c14d3eb-0adf-42fb-ba17-01de2938376f_characterizeImage_947523_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3c14d3eb-0adf-42fb-ba17-01de2938376f_characterizeImage_947523_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/691af0bd-3d3b-46c4-884b-fd8949fc9122_isr_947523_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/691af0bd-3d3b-46c4-884b-fd8949fc9122_isr_947523_31.stderr
parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24e72061678ceb42326c748cde7416b8 with result from task 312
parsl.dataflow.dflow DEBUG: Task 312 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 481cbc12298e63faac56f2f1b4c9d55e
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 515
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7d8709ba-fde6-4ba4-9c16-a768a61a9f92_characterizeImage_947420_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7d8709ba-fde6-4ba4-9c16-a768a61a9f92_characterizeImage_947420_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/933ad3e0-53d8-4d37-8a7c-742a9d8f3da5_isr_947420_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/933ad3e0-53d8-4d37-8a7c-742a9d8f3da5_isr_947420_56.stderr
parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e79ce336bf272608a5aac34a1c6f00fa with result from task 318
parsl.dataflow.dflow DEBUG: Task 318 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 86382ecc666890200c4766a0273ed8fe
parsl.dataflow.memoization INFO: Task 319 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 516
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057ad9f7-7420-4595-babe-f7c1853ff502_characterizeImage_947420_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057ad9f7-7420-4595-babe-f7c1853ff502_characterizeImage_947420_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d27d7185-225c-4dc4-9315-bde423fcf938_isr_947420_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d27d7185-225c-4dc4-9315-bde423fcf938_isr_947420_62.stderr
parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35db56a5f62d35e4c058bb242b1f77a4 with result from task 270
parsl.dataflow.dflow DEBUG: Task 270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 7f3d540157dc72a34a79caeed9d45671
parsl.dataflow.memoization INFO: Task 271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f004a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 517
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d9e00eb6-219b-4bd3-9523-c574f8ed66ec_characterizeImage_947418_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d9e00eb6-219b-4bd3-9523-c574f8ed66ec_characterizeImage_947418_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d3e162b3-7d8c-4cb2-8ab6-a244a59e4058_isr_947418_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d3e162b3-7d8c-4cb2-8ab6-a244a59e4058_isr_947418_29.stderr
parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ebef4deae6185a87aff16465017eb04 with result from task 246
parsl.dataflow.dflow DEBUG: Task 246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 247 has memoization hash a2c5dbd1374f692f5e234416f58644f2
parsl.dataflow.memoization INFO: Task 247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 518
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/56b97cb6-b815-466e-b5ce-8aafbc019464_characterizeImage_947420_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/56b97cb6-b815-466e-b5ce-8aafbc019464_characterizeImage_947420_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30320618-095e-46fa-a562-3fe85faf0da9_isr_947420_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30320618-095e-46fa-a562-3fe85faf0da9_isr_947420_23.stderr
parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 620533edd364fa989b469551ebc744fa with result from task 240
parsl.dataflow.dflow DEBUG: Task 240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 241 has memoization hash d8ff1e95a63d5b1e044dace157b107c9
parsl.dataflow.memoization INFO: Task 241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 519
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b54614e5-1f5d-48bc-9bcf-908696ad9ee1_characterizeImage_947420_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b54614e5-1f5d-48bc-9bcf-908696ad9ee1_characterizeImage_947420_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f4f899b-c0ff-4c8a-b953-bf335f9ac109_isr_947420_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f4f899b-c0ff-4c8a-b953-bf335f9ac109_isr_947420_46.stderr
parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6cc3b90cc7d684c3b70a436d668bc17 with result from task 243
parsl.dataflow.dflow DEBUG: Task 243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 8f1ddbd2a2de7153fb70729694a168be
parsl.dataflow.memoization INFO: Task 244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 520
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a2536188-88f4-4baf-8e21-da13ff19883e_characterizeImage_947422_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a2536188-88f4-4baf-8e21-da13ff19883e_characterizeImage_947422_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/1aaad9a1-4d74-4850-9bcf-a1f7dd44e2ed_isr_947422_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/1aaad9a1-4d74-4850-9bcf-a1f7dd44e2ed_isr_947422_36.stderr
parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ce8e1f697ed2e2c0189c17bcfb2c587 with result from task 231
parsl.dataflow.dflow DEBUG: Task 231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 4cc754335e7bd384afe6659c940f9e0c
parsl.dataflow.memoization INFO: Task 232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 521
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2bbdec7a-a2eb-4f34-9e46-42f07006d808_characterizeImage_947420_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2bbdec7a-a2eb-4f34-9e46-42f07006d808_characterizeImage_947420_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/23c85021-ddf3-4ae8-8f1c-e55603ea2dea_isr_947420_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/23c85021-ddf3-4ae8-8f1c-e55603ea2dea_isr_947420_36.stderr
parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e18572624e84aefb471e0f0a75b549c7 with result from task 228
parsl.dataflow.dflow DEBUG: Task 228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 229 has memoization hash c9dd2e2b64ee181c0f283e576102944e
parsl.dataflow.memoization INFO: Task 229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 522
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/69b9247c-afdc-4c08-bb30-1d9b8a367b77_characterizeImage_947418_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/69b9247c-afdc-4c08-bb30-1d9b8a367b77_characterizeImage_947418_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/95072ab7-8fbb-4a83-87c7-875b82e130dc_isr_947418_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/95072ab7-8fbb-4a83-87c7-875b82e130dc_isr_947418_1.stderr
parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e215bf2a12a57664149d0eeaac5be53 with result from task 234
parsl.dataflow.dflow DEBUG: Task 234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 16d7a580f2ab1d4cf8ab1a726c1b7744
parsl.dataflow.memoization INFO: Task 235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 523
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/93eaa5a6-8f48-4e06-aee5-c102880807d3_characterizeImage_947526_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/93eaa5a6-8f48-4e06-aee5-c102880807d3_characterizeImage_947526_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f1ed7b76-e25f-4b68-9dc5-d9a1a54229b8_isr_947526_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f1ed7b76-e25f-4b68-9dc5-d9a1a54229b8_isr_947526_4.stderr
parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 259f462893160238b91d8e94e3cfb21f with result from task 378
parsl.dataflow.dflow DEBUG: Task 378 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 414701d1aa3219f5ecf8009ed7e41fa4
parsl.dataflow.memoization INFO: Task 379 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 524
parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6682103b-48a3-433d-835f-559cd0235720_characterizeImage_947418_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6682103b-48a3-433d-835f-559cd0235720_characterizeImage_947418_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb533f57-2947-4a28-ab0c-13288f7ed778_isr_947418_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb533f57-2947-4a28-ab0c-13288f7ed778_isr_947418_13.stderr
parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a08a639dea99ce796931cdb2080755b4 with result from task 375
parsl.dataflow.dflow DEBUG: Task 375 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 900403fc1a81c391f790b1606dcdfdf0
parsl.dataflow.memoization INFO: Task 376 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 525
parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b4c64ea9-78e2-4331-a008-ab2d7a99ea7c_characterizeImage_947524_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b4c64ea9-78e2-4331-a008-ab2d7a99ea7c_characterizeImage_947524_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/cfe58559-c8ce-4124-98fe-b8e0ed65df7b_isr_947524_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/cfe58559-c8ce-4124-98fe-b8e0ed65df7b_isr_947524_58.stderr
parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fde419ef2456905b63e46753d94f8e9b with result from task 369
parsl.dataflow.dflow DEBUG: Task 369 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 987990a8f3c11e084c9eae7c88ffeead
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 526
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/10f2c386-3812-47ed-b3a1-3089a8c00f61_characterizeImage_947420_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/10f2c386-3812-47ed-b3a1-3089a8c00f61_characterizeImage_947420_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30a8834b-8fba-44fb-b982-9e4565c780e7_isr_947420_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/30a8834b-8fba-44fb-b982-9e4565c780e7_isr_947420_53.stderr
parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44992cf8cbf14256043223b2b678a920 with result from task 393
parsl.dataflow.dflow DEBUG: Task 393 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 77ec0113cebca15d7743be7dcc28dbdb
parsl.dataflow.memoization INFO: Task 394 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 527
parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/02196db1-baa5-415e-8637-dc4fa425611d_characterizeImage_947418_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/02196db1-baa5-415e-8637-dc4fa425611d_characterizeImage_947418_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c585f076-afec-4b32-b239-d81c89a4cf8b_isr_947418_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c585f076-afec-4b32-b239-d81c89a4cf8b_isr_947418_42.stderr
parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d284665a73e787da412fc8954ec8f09 with result from task 414
parsl.dataflow.dflow DEBUG: Task 414 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 415 has memoization hash e8462bb97902c51cfa02002f806ff987
parsl.dataflow.memoization INFO: Task 415 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 528
parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fd4b92dc-0d49-4325-9416-0f96d5f3af7a_characterizeImage_947526_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fd4b92dc-0d49-4325-9416-0f96d5f3af7a_characterizeImage_947526_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/73d7a956-273e-4057-a7b3-c186dc2286e5_isr_947526_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/73d7a956-273e-4057-a7b3-c186dc2286e5_isr_947526_44.stderr
parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6aac33d612fd9b4b38cfd7f78091cdac with result from task 402
parsl.dataflow.dflow DEBUG: Task 402 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 2d5147f8da71c9325c4b9c366ad63142
parsl.dataflow.memoization INFO: Task 403 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 529
parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8a0ef0ed-c4f5-49d9-8065-653d13cc1b2b_characterizeImage_947523_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8a0ef0ed-c4f5-49d9-8065-653d13cc1b2b_characterizeImage_947523_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0b14d0f5-cd5a-4954-9b21-5196a497bec9_isr_947523_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0b14d0f5-cd5a-4954-9b21-5196a497bec9_isr_947523_11.stderr
parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 303cc821a40fdec0cb75316cfeb28af1 with result from task 372
parsl.dataflow.dflow DEBUG: Task 372 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 373 has memoization hash f67452c2996569f4a88394b01260ceb8
parsl.dataflow.memoization INFO: Task 373 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 530
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6e403693-3e6a-4f0d-becd-5df99a83504a_isr_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6e403693-3e6a-4f0d-becd-5df99a83504a_isr_947528_2.stderr
parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ffbfe82670e5b77deb492409e413f9a with result from task 387
parsl.dataflow.dflow DEBUG: Task 387 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 388 has memoization hash da5af6ec86d42807fda4c81e1a0b00fd
parsl.dataflow.memoization INFO: Task 388 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 531
parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7c99a992-3049-41a4-b255-93cfee256cd3_characterizeImage_947420_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7c99a992-3049-41a4-b255-93cfee256cd3_characterizeImage_947420_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3492cd23-9da0-4650-8fd4-17579da37677_isr_947420_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3492cd23-9da0-4650-8fd4-17579da37677_isr_947420_42.stderr
parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f2b8c352111f37add915af2dc04a785 with result from task 384
parsl.dataflow.dflow DEBUG: Task 384 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 202d8dcaacb3a50a8b5f819840c0cf3d
parsl.dataflow.memoization INFO: Task 385 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 532
parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/cbbaf19e-5d13-4bd4-9fe0-fdd8749c8c28_characterizeImage_947422_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/cbbaf19e-5d13-4bd4-9fe0-fdd8749c8c28_characterizeImage_947422_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/61acb7f8-1fab-4385-b1c5-dfffaab3ecc2_isr_947422_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/61acb7f8-1fab-4385-b1c5-dfffaab3ecc2_isr_947422_54.stderr
parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0a563fb0fff2374d899dad8dc9cab58 with result from task 390
parsl.dataflow.dflow DEBUG: Task 390 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 7f3d8b4dddd6df9a4647663cb7a74f7d
parsl.dataflow.memoization INFO: Task 391 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 533
parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/65d669c7-3719-48ac-a2be-20bf562a9f7a_characterizeImage_947420_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/65d669c7-3719-48ac-a2be-20bf562a9f7a_characterizeImage_947420_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c60fd252-54c9-41ef-921b-5bb9e4ee9984_isr_947420_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c60fd252-54c9-41ef-921b-5bb9e4ee9984_isr_947420_22.stderr
parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 099bf0cedb54806d354de9ead9dfb29a with result from task 381
parsl.dataflow.dflow DEBUG: Task 381 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 382 has memoization hash b9c253c16b8d0341e3f5b3ad6301496f
parsl.dataflow.memoization INFO: Task 382 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 534
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/40e934cf-a02a-4b51-988b-e31872a1f52a_characterizeImage_947420_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/40e934cf-a02a-4b51-988b-e31872a1f52a_characterizeImage_947420_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47c4ffca-3c6b-49a9-aa71-938738fcdb8f_isr_947420_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47c4ffca-3c6b-49a9-aa71-938738fcdb8f_isr_947420_24.stderr
parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e74f9122e661ed5c71fea2b3e3315ecd with result from task 405
parsl.dataflow.dflow DEBUG: Task 405 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 9bf1125ddeabad3f93fd7f927c0e98aa
parsl.dataflow.memoization INFO: Task 406 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 535
parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0b18095a-87b6-437b-889a-0bea823485ce_characterizeImage_947422_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0b18095a-87b6-437b-889a-0bea823485ce_characterizeImage_947422_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/468b2559-d83b-4e10-8114-4fd6dd33d8db_isr_947422_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/468b2559-d83b-4e10-8114-4fd6dd33d8db_isr_947422_26.stderr
parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20bbe3894d9fd3af63e1251b48e33bc3 with result from task 408
parsl.dataflow.dflow DEBUG: Task 408 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 409 has memoization hash d0a0f347374850cdd4034a17fd7d4083
parsl.dataflow.memoization INFO: Task 409 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 536
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/e8f6f792-0ac3-4183-98c6-a05fba58b40d_characterizeImage_947526_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/e8f6f792-0ac3-4183-98c6-a05fba58b40d_characterizeImage_947526_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5ffd17d9-42d7-4446-88fd-e65f0cbec730_isr_947526_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5ffd17d9-42d7-4446-88fd-e65f0cbec730_isr_947526_28.stderr
parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd169110583857134280e70d1ba0eaae with result from task 396
parsl.dataflow.dflow DEBUG: Task 396 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 8b4835a49b4250348d9a1d17eaf5e248
parsl.dataflow.memoization INFO: Task 397 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 537
parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/53dc7927-573a-43af-8827-bf1ad43d23fd_characterizeImage_947420_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/53dc7927-573a-43af-8827-bf1ad43d23fd_characterizeImage_947420_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7e818045-3f79-40e7-aa11-5e1444cc01f8_isr_947420_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7e818045-3f79-40e7-aa11-5e1444cc01f8_isr_947420_34.stderr
parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27686ce769305482e7080c6e57800857 with result from task 411
parsl.dataflow.dflow DEBUG: Task 411 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 9c7f71dd9a04f20e9a636c92cc6f138e
parsl.dataflow.memoization INFO: Task 412 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 538
parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a3152ebd-8eac-498e-b1e3-84df30e6460e_characterizeImage_947523_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a3152ebd-8eac-498e-b1e3-84df30e6460e_characterizeImage_947523_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b1f2df3a-1d41-4d56-82f6-d985ca66bea7_isr_947523_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b1f2df3a-1d41-4d56-82f6-d985ca66bea7_isr_947523_20.stderr
parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e084bf7d12d7db9807c9d2bc6f216c8f with result from task 399
parsl.dataflow.dflow DEBUG: Task 399 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 01db9f60b67a797a655f9e99f4dac8ca
parsl.dataflow.memoization INFO: Task 400 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 539
parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5e07970d-16e9-42db-8767-66a80de058c9_characterizeImage_947523_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5e07970d-16e9-42db-8767-66a80de058c9_characterizeImage_947523_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/396b2ba5-a407-4335-862c-cfd4f6014fa0_isr_947523_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/396b2ba5-a407-4335-862c-cfd4f6014fa0_isr_947523_8.stderr
parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe18fda7c10f8113fe3bec4009ff1c6c with result from task 345
parsl.dataflow.dflow DEBUG: Task 345 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 6e41b9bc5eba37e19bd1c057e916c6b0
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 540
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d177ebed-29ad-42d1-a3d6-d2fe0d4ab53b_characterizeImage_947524_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d177ebed-29ad-42d1-a3d6-d2fe0d4ab53b_characterizeImage_947524_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/69d66532-c49f-408a-bfc3-3cc223b8a3a4_isr_947524_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/69d66532-c49f-408a-bfc3-3cc223b8a3a4_isr_947524_33.stderr
parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c7177f6c21dcad9e4171a6b7e41eae5 with result from task 348
parsl.dataflow.dflow DEBUG: Task 348 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 349 has memoization hash f971ca105259cea937a6131bb5cfc78f
parsl.dataflow.memoization INFO: Task 349 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 541
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9d9a7211-d59b-484b-81a5-849898877047_characterizeImage_947418_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9d9a7211-d59b-484b-81a5-849898877047_characterizeImage_947418_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d15e6e19-dca4-42c1-a914-86452e3591e7_isr_947418_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d15e6e19-dca4-42c1-a914-86452e3591e7_isr_947418_15.stderr
parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e124c872966de2eea98e049c80bc84e with result from task 300
parsl.dataflow.dflow DEBUG: Task 300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 301 has memoization hash ab488e4961886b898a1f176c2b76b8f0
parsl.dataflow.memoization INFO: Task 301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f023e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 542
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/771e82d6-4c13-4b18-b7c1-040e307c695b_characterizeImage_947526_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/771e82d6-4c13-4b18-b7c1-040e307c695b_characterizeImage_947526_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/56732e0a-bf73-4ad4-8d6d-0b21a76a507b_isr_947526_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/56732e0a-bf73-4ad4-8d6d-0b21a76a507b_isr_947526_12.stderr
parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db50166195a414d6c51f9b832546878c with result from task 315
parsl.dataflow.dflow DEBUG: Task 315 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 8c94770ac45dd696a4adb00b74a1cb2c
parsl.dataflow.memoization INFO: Task 316 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 543
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d27506b9-ca57-4c5e-b64c-5c49fde95470_characterizeImage_947526_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d27506b9-ca57-4c5e-b64c-5c49fde95470_characterizeImage_947526_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e9e62966-cb3d-496a-9ecb-eb7737439bf9_isr_947526_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e9e62966-cb3d-496a-9ecb-eb7737439bf9_isr_947526_6.stderr
parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7182c5a3b02e67adfc611b9415e1d296 with result from task 309
parsl.dataflow.dflow DEBUG: Task 309 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash c843e496c468799f4b0795d4d6399bf1
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 544
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6d4806ce-a683-4104-95c7-cf32e3caafee_characterizeImage_947418_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6d4806ce-a683-4104-95c7-cf32e3caafee_characterizeImage_947418_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/20fe30b2-8624-42b5-b48d-2e0c6841d1cc_isr_947418_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/20fe30b2-8624-42b5-b48d-2e0c6841d1cc_isr_947418_3.stderr
parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7f612d38e1ef22ab92de24bf548bd43 with result from task 297
parsl.dataflow.dflow DEBUG: Task 297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 5581209b615ffc3498f7e1084ca81e57
parsl.dataflow.memoization INFO: Task 298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f020c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 545
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/254358e0-3a00-4887-be59-8258df35c665_characterizeImage_947526_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/254358e0-3a00-4887-be59-8258df35c665_characterizeImage_947526_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/41cf2400-f503-4fdb-89e5-403643656846_isr_947526_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/41cf2400-f503-4fdb-89e5-403643656846_isr_947526_55.stderr
parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f7a6be46a8efc2054091f6efb353b35 with result from task 1071
parsl.dataflow.dflow DEBUG: Task 1071 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash ae560dab7dbbdb2c79b17ef40028d640
parsl.dataflow.memoization INFO: Task 1072 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 546
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8200a910-e4eb-4a0f-a8fe-6829c982221f_characterizeImage_947524_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8200a910-e4eb-4a0f-a8fe-6829c982221f_characterizeImage_947524_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/44ac5511-67eb-4236-aac8-3431a8930868_isr_947524_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/44ac5511-67eb-4236-aac8-3431a8930868_isr_947524_22.stderr
parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed013c4a6d237e68a93f60ea9e20aec8 with result from task 363
parsl.dataflow.dflow DEBUG: Task 363 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 2f802e52cc135b3b784bc5a178d68319
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 547
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/327dd411-0946-4cb1-bf57-73fbc1d98353_characterizeImage_947420_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/327dd411-0946-4cb1-bf57-73fbc1d98353_characterizeImage_947420_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/38b1a1eb-7105-40d1-91f3-f5fd08a041db_isr_947420_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/38b1a1eb-7105-40d1-91f3-f5fd08a041db_isr_947420_30.stderr
parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 865a97b2466c858f4dc95bbf9129b919 with result from task 360
parsl.dataflow.dflow DEBUG: Task 360 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 3f8179b7405c34125891758994c2388d
parsl.dataflow.memoization INFO: Task 361 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 548
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2860d712-3a21-463f-8936-58523f08b9bc_characterizeImage_947420_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2860d712-3a21-463f-8936-58523f08b9bc_characterizeImage_947420_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f0328ea-977b-49d1-85ed-a941486f59c8_isr_947420_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5f0328ea-977b-49d1-85ed-a941486f59c8_isr_947420_11.stderr
parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4525d1f36eaa6bc23e1eec455304db64 with result from task 357
parsl.dataflow.dflow DEBUG: Task 357 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 358 has memoization hash b8adbb92919af5f34792512264373dad
parsl.dataflow.memoization INFO: Task 358 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 549
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/1d6ea37f-bf3f-401d-a72e-beb25614b886_characterizeImage_947523_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/1d6ea37f-bf3f-401d-a72e-beb25614b886_characterizeImage_947523_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7667c897-246d-4f9f-9303-0c8388ac2b29_isr_947523_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7667c897-246d-4f9f-9303-0c8388ac2b29_isr_947523_42.stderr
parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6d12ffa4ffa7b79d4b56e72b4890df5 with result from task 366
parsl.dataflow.dflow DEBUG: Task 366 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 367 has memoization hash e2d57eabeb13e3079c39afdfab15737e
parsl.dataflow.memoization INFO: Task 367 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 550
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/926ec619-c793-4970-9670-bd688ad779bf_characterizeImage_947528_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/926ec619-c793-4970-9670-bd688ad779bf_characterizeImage_947528_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7fd26538-ba62-4222-ab52-8466592e578f_isr_947528_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/7fd26538-ba62-4222-ab52-8466592e578f_isr_947528_54.stderr
parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9f6a6e679273cddece14ee69527b0d3d with result from task 354
parsl.dataflow.dflow DEBUG: Task 354 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash b9e1c6627a601be48a157fd57f4bb5fa
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 551
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/97a08412-8d7a-44f2-b603-e328921b758e_characterizeImage_947526_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/97a08412-8d7a-44f2-b603-e328921b758e_characterizeImage_947526_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/589bc693-3130-4169-8e85-24ce137a9cd5_isr_947526_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/589bc693-3130-4169-8e85-24ce137a9cd5_isr_947526_54.stderr
parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afb4636fcb6049e9c8b2c4fdb6a61151 with result from task 435
parsl.dataflow.dflow DEBUG: Task 435 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 3e6a2769da80966cade315178a1e7ec9
parsl.dataflow.memoization INFO: Task 436 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 552
parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8cb46067-cb45-4770-83ee-ffbc539481ee_characterizeImage_947524_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8cb46067-cb45-4770-83ee-ffbc539481ee_characterizeImage_947524_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/a722a0e6-3790-4aca-ab2a-c281e41d538c_isr_947524_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/a722a0e6-3790-4aca-ab2a-c281e41d538c_isr_947524_15.stderr
parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14750067423c24d5525b7f4da21130f1 with result from task 420
parsl.dataflow.dflow DEBUG: Task 420 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 421 has memoization hash b6ec749f479cadd5645b4e7a476ba3cc
parsl.dataflow.memoization INFO: Task 421 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 553
parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/23c1c262-ea5b-40f1-844b-74def6b95e13_characterizeImage_947418_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/23c1c262-ea5b-40f1-844b-74def6b95e13_characterizeImage_947418_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a44530df-6d8a-4803-9445-379becfd8990_isr_947418_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a44530df-6d8a-4803-9445-379becfd8990_isr_947418_32.stderr
parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry feadaa1d6af92f8282decfcc2e19f469 with result from task 417
parsl.dataflow.dflow DEBUG: Task 417 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 418 has memoization hash a37380b2574cff72fb7b437a24af852e
parsl.dataflow.memoization INFO: Task 418 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 554
parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/313bd249-64ae-4f36-8fb3-6073fcf1663d_characterizeImage_947528_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/313bd249-64ae-4f36-8fb3-6073fcf1663d_characterizeImage_947528_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b6e21d70-5f0c-431a-9493-37bc701ad0ca_isr_947528_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b6e21d70-5f0c-431a-9493-37bc701ad0ca_isr_947528_32.stderr
parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be04bc1796e93f2fa898adc690210f4d with result from task 321
parsl.dataflow.dflow DEBUG: Task 321 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 005332694cc1a6224320071526c34715
parsl.dataflow.memoization INFO: Task 322 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 555
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9a75a310-4d85-49b4-8532-edb4991df967_characterizeImage_947420_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9a75a310-4d85-49b4-8532-edb4991df967_characterizeImage_947420_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4a54b160-cdec-46c0-bb1a-3b7dba461183_isr_947420_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4a54b160-cdec-46c0-bb1a-3b7dba461183_isr_947420_51.stderr
parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 624ffe445aabce00c97c4b2b04be5542 with result from task 330
parsl.dataflow.dflow DEBUG: Task 330 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash decd62d1ff6f00ef7fb36f0b1e6c57c2
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 556
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebf3b7b5-278a-4b02-9b07-9dab6bb345a5_characterizeImage_947422_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebf3b7b5-278a-4b02-9b07-9dab6bb345a5_characterizeImage_947422_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5573ebb0-648e-4e74-a055-457b4079740f_isr_947422_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5573ebb0-648e-4e74-a055-457b4079740f_isr_947422_8.stderr
parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1dce8b1920247ee8540f2bb2d399947 with result from task 423
parsl.dataflow.dflow DEBUG: Task 423 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 424 has memoization hash e47dbc82ec06aad4d1e7caa248ab894f
parsl.dataflow.memoization INFO: Task 424 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 557
parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/02365590-da4e-44ac-8d8d-48404ac57a09_characterizeImage_947422_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/02365590-da4e-44ac-8d8d-48404ac57a09_characterizeImage_947422_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/472b2159-067a-4055-a02a-f402c27d6729_isr_947422_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/472b2159-067a-4055-a02a-f402c27d6729_isr_947422_45.stderr
parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e83ca24da964101c455697be7298ec17 with result from task 432
parsl.dataflow.dflow DEBUG: Task 432 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 433 has memoization hash c9665f8c9904b41ed1a73967870f4870
parsl.dataflow.memoization INFO: Task 433 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 558
parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ea78427a-da0d-465d-b267-4486e6194623_characterizeImage_947420_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ea78427a-da0d-465d-b267-4486e6194623_characterizeImage_947420_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8b43018e-e111-406e-8936-4421ae7ac670_isr_947420_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8b43018e-e111-406e-8936-4421ae7ac670_isr_947420_45.stderr
2024-03-14 07:26:46 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
parsl.dataflow.dflow INFO: Task 1269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14c35b8f8d8f26d799d611d633ba3190 with result from task 1269
parsl.dataflow.dflow DEBUG: Task 1269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1270 has memoization hash 450bdf9bb809d6a1ea991e0c48a2f801
parsl.dataflow.memoization INFO: Task 1270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1270 try 0 launched on executor multi with executor id 559
parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4775f391-f7ca-4b61-8fda-a66f8e2b77b5_characterizeImage_947422_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4775f391-f7ca-4b61-8fda-a66f8e2b77b5_characterizeImage_947422_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9660d08-1d46-44cb-b3a8-f6085b79b8b4_isr_947422_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9660d08-1d46-44cb-b3a8-f6085b79b8b4_isr_947422_23.stderr
parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 320c6fded4fe80a5fb49c4d1993e2f88 with result from task 342
parsl.dataflow.dflow DEBUG: Task 342 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 343 has memoization hash c5ba59d66af99c89430cdd3a25ecd4ca
parsl.dataflow.memoization INFO: Task 343 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 560
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/55c08191-dc62-4879-90c4-ed0d9042643e_characterizeImage_947418_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/55c08191-dc62-4879-90c4-ed0d9042643e_characterizeImage_947418_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1c767594-9473-446d-b06b-287f51c8449a_isr_947418_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1c767594-9473-446d-b06b-287f51c8449a_isr_947418_25.stderr
2024-03-14 07:26:46 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60b08279625e05391926f02474447c4c with result from task 1170
parsl.dataflow.dflow DEBUG: Task 1170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1171 has memoization hash 2823ea96bbcac6c6b0f3ba4d4ecdee37
parsl.dataflow.memoization INFO: Task 1171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1171 try 0 launched on executor multi with executor id 561
parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/98638c71-64f9-4b2e-b85c-1f112c2b3ed6_characterizeImage_947526_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/98638c71-64f9-4b2e-b85c-1f112c2b3ed6_characterizeImage_947526_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5bba55d6-93c9-480f-a742-a5044daabc08_isr_947526_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5bba55d6-93c9-480f-a742-a5044daabc08_isr_947526_43.stderr
parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d95e82b09f7744811cc5d8b892610a7d with result from task 324
parsl.dataflow.dflow DEBUG: Task 324 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 74964db2664efd20f188564134a82f85
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 562
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/aa2ea0c8-68b1-4ec7-b179-a946b1d3e0e0_characterizeImage_947523_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/aa2ea0c8-68b1-4ec7-b179-a946b1d3e0e0_characterizeImage_947523_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7b6167b8-e550-4de5-97b9-7fa01051b86e_isr_947523_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7b6167b8-e550-4de5-97b9-7fa01051b86e_isr_947523_30.stderr
parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0aa2ee49a09133fee6cc1f3adddb3fd with result from task 327
parsl.dataflow.dflow DEBUG: Task 327 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 2bfbf969e94699fd1e6cbf78a867547c
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 563
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/76d545f7-4666-4e62-8134-8f79449df4d5_characterizeImage_947418_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/76d545f7-4666-4e62-8134-8f79449df4d5_characterizeImage_947418_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/66a527a8-1cc4-4a5e-841c-a219517274af_isr_947418_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/66a527a8-1cc4-4a5e-841c-a219517274af_isr_947418_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:46 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d7f1fb8ed553fc6f256eac103cbaa72 with result from task 1296
parsl.dataflow.dflow DEBUG: Task 1296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1297 has memoization hash cd6cb09c73e86d79abb33e2e2694da8c
parsl.dataflow.memoization INFO: Task 1297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1297 try 0 launched on executor multi with executor id 564
parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/691293ff-1a55-490c-ab10-49bfea162f0f_characterizeImage_947523_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/691293ff-1a55-490c-ab10-49bfea162f0f_characterizeImage_947523_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6eba221a-d014-4720-82eb-1d8e8dea66d5_isr_947523_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6eba221a-d014-4720-82eb-1d8e8dea66d5_isr_947523_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:47 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c92b6fa14b462286ae89677fde510da6 with result from task 333
parsl.dataflow.dflow DEBUG: Task 333 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 135d9c29b27eaa0def58668460e00577
parsl.dataflow.memoization INFO: Task 334 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 565
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/97091f58-64f8-4dd4-afda-997df0c34393_characterizeImage_947524_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/97091f58-64f8-4dd4-afda-997df0c34393_characterizeImage_947524_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/feb4d085-3c17-4639-beac-dc3570e835c4_isr_947524_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/feb4d085-3c17-4639-beac-dc3570e835c4_isr_947524_34.stderr
parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae56f8d158ecd2b470ed19f49e6c741e with result from task 339
parsl.dataflow.dflow DEBUG: Task 339 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 340 has memoization hash b3914b636e1f267d2163d4aff10be67a
parsl.dataflow.memoization INFO: Task 340 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 566
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a2ab6cfb-2e8c-4e4b-8273-904d25d54f0c_characterizeImage_947524_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a2ab6cfb-2e8c-4e4b-8273-904d25d54f0c_characterizeImage_947524_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/71e3df41-3bca-471c-9705-61297254aef0_isr_947524_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/71e3df41-3bca-471c-9705-61297254aef0_isr_947524_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:47 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad4f12b82457444c988ef5d4bc249743 with result from task 426
parsl.dataflow.dflow DEBUG: Task 426 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 69697bdaaf51446de99ea29b26ac8bfa
parsl.dataflow.memoization INFO: Task 427 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f527a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 567
parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/cd3408c0-bcf7-4bd1-955e-6c4b7afcf3ff_characterizeImage_947420_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/cd3408c0-bcf7-4bd1-955e-6c4b7afcf3ff_characterizeImage_947420_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/943e6b8b-5d28-4d27-92b6-bf0f6d63a08f_isr_947420_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/943e6b8b-5d28-4d27-92b6-bf0f6d63a08f_isr_947420_3.stderr
parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ff3113b98e3ee0b40591c6c2a01b769 with result from task 438
parsl.dataflow.dflow DEBUG: Task 438 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 439 has memoization hash f9d86ead6ad9dd43a37e8f81dcfc9f83
parsl.dataflow.memoization INFO: Task 439 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 568
parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/72d3b1bd-bf14-4e55-8942-0a0b855788de_characterizeImage_947526_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/72d3b1bd-bf14-4e55-8942-0a0b855788de_characterizeImage_947526_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e260a333-e212-48d2-91d7-97c335ed633f_isr_947526_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e260a333-e212-48d2-91d7-97c335ed633f_isr_947526_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:47 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9d51f620f04eacad437278305ebd3dc with result from task 336
parsl.dataflow.dflow DEBUG: Task 336 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 57cf70d01296b359f0c1c75227844f49
parsl.dataflow.memoization INFO: Task 337 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 569
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ecf6aed4-bc89-4593-b881-abd94961dc18_characterizeImage_947418_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ecf6aed4-bc89-4593-b881-abd94961dc18_characterizeImage_947418_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bce8f3f3-e731-47d9-88bb-143ef16fe8e1_isr_947418_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bce8f3f3-e731-47d9-88bb-143ef16fe8e1_isr_947418_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:47 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 515c68fef976624118c6b966bf3237cf with result from task 1299
parsl.dataflow.dflow DEBUG: Task 1299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1300 has memoization hash cb9ae21bfec2233ba66067068d7ea276
parsl.dataflow.memoization INFO: Task 1300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1300 try 0 launched on executor multi with executor id 570
parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cf0725c3-d177-47a8-833c-1f3c7723a383_characterizeImage_947418_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cf0725c3-d177-47a8-833c-1f3c7723a383_characterizeImage_947418_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2e0e3ed0-6f13-452d-bb5d-b5a7dde87686_isr_947418_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2e0e3ed0-6f13-452d-bb5d-b5a7dde87686_isr_947418_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 612aaa34eb289262b1d84e240d029d29 with result from task 444
parsl.dataflow.dflow DEBUG: Task 444 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 9227d04482233a40557dea419d9aeb0a
parsl.dataflow.memoization INFO: Task 445 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 571
parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7f8f9ddd-c04f-4403-8367-fa1bf33c85c1_characterizeImage_947422_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7f8f9ddd-c04f-4403-8367-fa1bf33c85c1_characterizeImage_947422_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7db810a3-a9bb-46c6-92a6-36a5198913c4_isr_947422_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7db810a3-a9bb-46c6-92a6-36a5198913c4_isr_947422_46.stderr
parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2c220f4923c2bce3a36abe1e95422bc with result from task 462
parsl.dataflow.dflow DEBUG: Task 462 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 7c5966f6ff8cc44494196fcb0735c4ea
parsl.dataflow.memoization INFO: Task 463 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 572
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/0d840017-7316-4b85-8026-b32f8e6e13d6_characterizeImage_947418_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/0d840017-7316-4b85-8026-b32f8e6e13d6_characterizeImage_947418_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cd8a868-2e99-4de7-9064-815c240abb3a_isr_947418_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cd8a868-2e99-4de7-9064-815c240abb3a_isr_947418_43.stderr
parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7e45bcee568158002ccea095cb568c5 with result from task 456
parsl.dataflow.dflow DEBUG: Task 456 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 457 has memoization hash c72a43c1090f7146c77c20b393931bce
parsl.dataflow.memoization INFO: Task 457 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 573
parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2e30590f-4f5b-4fae-8a2a-643e03aec06e_characterizeImage_947422_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2e30590f-4f5b-4fae-8a2a-643e03aec06e_characterizeImage_947422_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/4f3173f8-1b27-43bf-9839-f61007b18c51_isr_947422_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/4f3173f8-1b27-43bf-9839-f61007b18c51_isr_947422_10.stderr
parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 833d202ecc71fca1447a9097a5181c3e with result from task 459
parsl.dataflow.dflow DEBUG: Task 459 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 460 has memoization hash d39bf69ddac91056d1e1921961a7db26
parsl.dataflow.memoization INFO: Task 460 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 574
parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2637e1be-a477-4e5f-b747-8317cbc909af_characterizeImage_947526_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2637e1be-a477-4e5f-b747-8317cbc909af_characterizeImage_947526_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/31a82a45-849a-4e38-9b2b-1238e594f89d_isr_947526_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/31a82a45-849a-4e38-9b2b-1238e594f89d_isr_947526_25.stderr
parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88970bf204c268c34c8dfcc068b01ce5 with result from task 441
parsl.dataflow.dflow DEBUG: Task 441 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 442 has memoization hash d9de9b6b77ff3cf1320d9ba584551163
parsl.dataflow.memoization INFO: Task 442 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 575
parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2bc03435-3008-4dce-aae6-802c998ee50b_characterizeImage_947528_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2bc03435-3008-4dce-aae6-802c998ee50b_characterizeImage_947528_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f7428d9c-2995-43ca-8c2c-dd2da17ccb7b_isr_947528_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f7428d9c-2995-43ca-8c2c-dd2da17ccb7b_isr_947528_21.stderr
parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32a74d13485f94bf315296079b932c6c with result from task 450
parsl.dataflow.dflow DEBUG: Task 450 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 3dabb34167ba7b9f1304f5379f154a5a
parsl.dataflow.memoization INFO: Task 451 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 576
parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a5ac77f6-c243-4a16-80bf-6049d8759f92_characterizeImage_947420_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a5ac77f6-c243-4a16-80bf-6049d8759f92_characterizeImage_947420_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e8ad685-80e6-4e93-a13f-cacd1efef590_isr_947420_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e8ad685-80e6-4e93-a13f-cacd1efef590_isr_947420_48.stderr
parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cde68f7becaaa1cb60b63ec435eec9a with result from task 447
parsl.dataflow.dflow DEBUG: Task 447 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 4ca186ada87c983c95b26e9453509ccf
parsl.dataflow.memoization INFO: Task 448 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 577
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b6da41f9-9d30-44b0-95cf-c14030a80468_isr_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b6da41f9-9d30-44b0-95cf-c14030a80468_isr_947422_61.stderr
parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c00b551246051213f5912cacc25fa6bc with result from task 453
parsl.dataflow.dflow DEBUG: Task 453 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 5de0a64f42183bdc41c2f02ed1e5f6c6
parsl.dataflow.memoization INFO: Task 454 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 578
parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4972949c-a6f2-4f1c-bfa5-96f74d2c73f9_characterizeImage_947523_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4972949c-a6f2-4f1c-bfa5-96f74d2c73f9_characterizeImage_947523_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c5a899d-a38f-40ba-ac43-a10c2a684460_isr_947523_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c5a899d-a38f-40ba-ac43-a10c2a684460_isr_947523_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:48 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2887590a292bc3d2f14c350a18b67cc with result from task 792
parsl.dataflow.dflow DEBUG: Task 792 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 5b50a48ae4c7e896f9be9ca8b9882cca
parsl.dataflow.memoization INFO: Task 793 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 579
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/07014230-0fb5-431f-a72b-09a357cea158_characterizeImage_947422_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/07014230-0fb5-431f-a72b-09a357cea158_characterizeImage_947422_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f8d8d229-bc52-467a-86fa-8b6cbd4d5ef4_isr_947422_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f8d8d229-bc52-467a-86fa-8b6cbd4d5ef4_isr_947422_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73c5ff22c9cb2f9abbb3a851a7ab49d8 with result from task 429
parsl.dataflow.dflow DEBUG: Task 429 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 430 has memoization hash abaa6c75c326fc757e3f6254f350704a
parsl.dataflow.memoization INFO: Task 430 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 580
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/43041829-f398-49cb-a705-ed43cb8aa5f0_characterizeImage_947422_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/43041829-f398-49cb-a705-ed43cb8aa5f0_characterizeImage_947422_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/9f03f79b-c880-4a5f-8aaa-a3ae442016db_isr_947422_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/9f03f79b-c880-4a5f-8aaa-a3ae442016db_isr_947422_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:49 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25b612b0302d704324ca72e7b4918378 with result from task 795
parsl.dataflow.dflow DEBUG: Task 795 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 796 has memoization hash d2b33e1abc14cba1bb337e451e585a0f
parsl.dataflow.memoization INFO: Task 796 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 581
parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a0a77c70-b8b5-4245-a950-be787fcb9f95_characterizeImage_947420_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a0a77c70-b8b5-4245-a950-be787fcb9f95_characterizeImage_947420_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9002e6c0-24f9-4a1a-ad26-407bb72e0235_isr_947420_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9002e6c0-24f9-4a1a-ad26-407bb72e0235_isr_947420_14.stderr
parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 537dee68b50e8cc3fd28399a8c367626 with result from task 801
parsl.dataflow.dflow DEBUG: Task 801 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 802 has memoization hash b74634238c16a9193c1281b9a1fdbecc
parsl.dataflow.memoization INFO: Task 802 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 582
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/86daac64-0c72-497c-877f-bfac49bc9253_characterizeImage_947524_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/86daac64-0c72-497c-877f-bfac49bc9253_characterizeImage_947524_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e58e469-e681-4adc-850d-26c38accfc64_isr_947524_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e58e469-e681-4adc-850d-26c38accfc64_isr_947524_1.stderr
parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5fcd80facd22767af857986afd471648 with result from task 786
parsl.dataflow.dflow DEBUG: Task 786 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 09297afee3905e1674b11fbd2fc6be94
parsl.dataflow.memoization INFO: Task 787 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 583
parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b9af7cab-ac59-4341-b232-a6372f924947_characterizeImage_947524_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b9af7cab-ac59-4341-b232-a6372f924947_characterizeImage_947524_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5d1d7ba8-0715-40c1-b0c8-e5f2406facd3_isr_947524_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5d1d7ba8-0715-40c1-b0c8-e5f2406facd3_isr_947524_16.stderr
parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba494717c3189ccafab75c2a9e6ab8ed with result from task 783
parsl.dataflow.dflow DEBUG: Task 783 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 3af06f42209a7112bf486d4514834fc0
parsl.dataflow.memoization INFO: Task 784 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 584
parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fbc98c96-7f9d-4a03-98a7-a20f660266b3_characterizeImage_947523_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fbc98c96-7f9d-4a03-98a7-a20f660266b3_characterizeImage_947523_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/89d89f54-ae01-42d6-b2be-d0fbe877dbea_isr_947523_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/89d89f54-ae01-42d6-b2be-d0fbe877dbea_isr_947523_49.stderr
parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fcad45cd2ad4a6a902b24e0dfd9b0728 with result from task 804
parsl.dataflow.dflow DEBUG: Task 804 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 42184812652284d6751eb214d5b2539a
parsl.dataflow.memoization INFO: Task 805 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 585
parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/50b857ff-8b99-48bc-9216-7ea131e5413c_characterizeImage_947528_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/50b857ff-8b99-48bc-9216-7ea131e5413c_characterizeImage_947528_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c472aede-cfaa-4cce-92c7-f23d43adace6_isr_947528_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c472aede-cfaa-4cce-92c7-f23d43adace6_isr_947528_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 993204158e2beff107612086a8f39774 with result from task 798
parsl.dataflow.dflow DEBUG: Task 798 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 799 has memoization hash b46d32649a7db629377d7b314d815e23
parsl.dataflow.memoization INFO: Task 799 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 586
parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cee2c7f7-580f-43df-a92d-02aba249eff6_characterizeImage_947524_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cee2c7f7-580f-43df-a92d-02aba249eff6_characterizeImage_947524_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6c04b92e-7a9f-4932-955f-cb6902014982_isr_947524_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6c04b92e-7a9f-4932-955f-cb6902014982_isr_947524_55.stderr
parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d2ee007bbdfbae2a6408dc4d7590743 with result from task 789
parsl.dataflow.dflow DEBUG: Task 789 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 09cd6f07cceb67bbd49ce16cc0411010
parsl.dataflow.memoization INFO: Task 790 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 587
parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/656be7ee-cac3-41ce-afdf-39177977a93b_characterizeImage_947422_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/656be7ee-cac3-41ce-afdf-39177977a93b_characterizeImage_947422_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/390c5d23-6fb0-4c98-845e-8042df6b6e27_isr_947422_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/390c5d23-6fb0-4c98-845e-8042df6b6e27_isr_947422_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f891459501c74a70220f19f8c58455fe with result from task 483
parsl.dataflow.dflow DEBUG: Task 483 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 484 has memoization hash deebd9ac87f689aa6d56671d51778263
parsl.dataflow.memoization INFO: Task 484 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 588
parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/75ec388f-6f50-4281-878a-8c75eb1ca030_characterizeImage_947528_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/75ec388f-6f50-4281-878a-8c75eb1ca030_characterizeImage_947528_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/5c758297-6fd7-475c-a5a8-f194e9a68ff3_isr_947528_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/5c758297-6fd7-475c-a5a8-f194e9a68ff3_isr_947528_53.stderr
parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry edf6917f329c85af1451b45049132054 with result from task 477
parsl.dataflow.dflow DEBUG: Task 477 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 40810a8d2f736ec28772ab5530a906eb
parsl.dataflow.memoization INFO: Task 478 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 589
parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af266b67-207e-49a1-96a3-fe39a6c061c3_characterizeImage_947528_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af266b67-207e-49a1-96a3-fe39a6c061c3_characterizeImage_947528_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8d0e1e5b-9133-44f9-b7d7-98212a7422df_isr_947528_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8d0e1e5b-9133-44f9-b7d7-98212a7422df_isr_947528_59.stderr
parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc5f236e6332122fa968136dae4c40c with result from task 468
parsl.dataflow.dflow DEBUG: Task 468 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 6809fbd0b4e460619b07fb9e75f2c215
parsl.dataflow.memoization INFO: Task 469 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 590
parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a475c957-2a9c-42fc-bbe5-31c7b616f448_characterizeImage_947420_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a475c957-2a9c-42fc-bbe5-31c7b616f448_characterizeImage_947420_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e1957a0-c51d-4ba6-a9bb-d849c2dc2d4c_isr_947420_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/0e1957a0-c51d-4ba6-a9bb-d849c2dc2d4c_isr_947420_26.stderr
parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d746b1a5ae4039545533243a11fc28b2 with result from task 474
parsl.dataflow.dflow DEBUG: Task 474 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 475 has memoization hash a8e3c2325174dde0fd7c38430b782b61
parsl.dataflow.memoization INFO: Task 475 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 591
parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fa768a72-23b4-4b72-87fe-db12ca0e200b_characterizeImage_947526_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fa768a72-23b4-4b72-87fe-db12ca0e200b_characterizeImage_947526_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1921d2ea-b283-4ebe-a684-c5f5a10a74c1_isr_947526_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1921d2ea-b283-4ebe-a684-c5f5a10a74c1_isr_947526_11.stderr
parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6672dad07a435243c87bdfce911dd9fc with result from task 489
parsl.dataflow.dflow DEBUG: Task 489 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 51f634817e48dded9ca043615ba7615d
parsl.dataflow.memoization INFO: Task 490 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 592
parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a781f214-5241-47ae-a078-c5a8fbbca4d2_characterizeImage_947523_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a781f214-5241-47ae-a078-c5a8fbbca4d2_characterizeImage_947523_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1f06fbd2-1c75-462a-8826-693feb81b9b1_isr_947523_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1f06fbd2-1c75-462a-8826-693feb81b9b1_isr_947523_9.stderr
parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0a3e651b5f7ce3acc7527283a5ce25f with result from task 486
parsl.dataflow.dflow DEBUG: Task 486 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 0723951560651cf84d35e422a963bd96
parsl.dataflow.memoization INFO: Task 487 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 593
parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebb751ff-fadd-47c6-bf73-bdbe34f81d86_characterizeImage_947422_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebb751ff-fadd-47c6-bf73-bdbe34f81d86_characterizeImage_947422_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/e50fa31f-bede-4b47-b200-5d202a6c3085_isr_947422_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/e50fa31f-bede-4b47-b200-5d202a6c3085_isr_947422_35.stderr
parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry baa45e2ce6041a2a613129d1f0e95b8d with result from task 564
parsl.dataflow.dflow DEBUG: Task 564 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 34f0c061a94b7de311bd61c0e7b60366
parsl.dataflow.memoization INFO: Task 565 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 594
parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bae9f807-679f-4a81-b1f7-6403fffb733d_characterizeImage_947526_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bae9f807-679f-4a81-b1f7-6403fffb733d_characterizeImage_947526_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/34b8a893-2d25-4cdd-81ac-0e94bd12e5bc_isr_947526_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/34b8a893-2d25-4cdd-81ac-0e94bd12e5bc_isr_947526_59.stderr
parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 721526aa5e820a3ec519fe649987464d with result from task 570
parsl.dataflow.dflow DEBUG: Task 570 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 571 has memoization hash b0775dd69de2f079c9945ce68d66c365
parsl.dataflow.memoization INFO: Task 571 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 595
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4aef455a-325e-4ac3-aa13-c609ee59a344_characterizeImage_947420_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4aef455a-325e-4ac3-aa13-c609ee59a344_characterizeImage_947420_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5782d2bd-150e-45ed-b08c-c584aa4f5d55_isr_947420_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5782d2bd-150e-45ed-b08c-c584aa4f5d55_isr_947420_16.stderr
parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c914b2424013f0edbb28e552cd79a8d with result from task 576
parsl.dataflow.dflow DEBUG: Task 576 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 577 has memoization hash e989471fe96c4b1694c198ba66dc3a38
parsl.dataflow.memoization INFO: Task 577 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:50 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 596
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7d05f873-6205-468c-9904-559d7ad63e8f_characterizeImage_947523_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7d05f873-6205-468c-9904-559d7ad63e8f_characterizeImage_947523_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6513be22-6ed4-40f0-a836-8f6dd03a8551_isr_947523_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6513be22-6ed4-40f0-a836-8f6dd03a8551_isr_947523_23.stderr
parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7c6590b8d90f66b5883296ecd6ead7e with result from task 579
parsl.dataflow.dflow DEBUG: Task 579 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 580 has memoization hash bcf492929a8ba2c3ffad63d716f5b72d
parsl.dataflow.memoization INFO: Task 580 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 597
parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2d184a87-f8f9-4920-8798-183701062ae3_characterizeImage_947420_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2d184a87-f8f9-4920-8798-183701062ae3_characterizeImage_947420_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/35120db0-0364-4cd2-818c-cd24480ee67e_isr_947420_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/35120db0-0364-4cd2-818c-cd24480ee67e_isr_947420_55.stderr
parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b4ca368e2c47d4d07addfac6490c44b with result from task 717
parsl.dataflow.dflow DEBUG: Task 717 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 718 has memoization hash d59ade1db57b55feb186d2ab89d3c162
parsl.dataflow.memoization INFO: Task 718 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 598
parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/891e1966-fd80-4ca1-a618-dbc7cea903f2_characterizeImage_947420_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/891e1966-fd80-4ca1-a618-dbc7cea903f2_characterizeImage_947420_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/43adf152-7201-4d10-ac4e-6b91eb464880_isr_947420_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/43adf152-7201-4d10-ac4e-6b91eb464880_isr_947420_15.stderr
parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 198cc807805b1e5ae08d94d631a9c740 with result from task 723
parsl.dataflow.dflow DEBUG: Task 723 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 724 has memoization hash bd38cf2d63febb8243bd345f9cc1cdc9
parsl.dataflow.memoization INFO: Task 724 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 599
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c2622d53-2605-4459-abc3-a54f887bf020_characterizeImage_947526_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c2622d53-2605-4459-abc3-a54f887bf020_characterizeImage_947526_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f2e3aa78-c652-4708-9a49-0ace4d7078af_isr_947526_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f2e3aa78-c652-4708-9a49-0ace4d7078af_isr_947526_53.stderr
parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d1d3ef1891a9a5836a75b49bf2512e5 with result from task 720
parsl.dataflow.dflow DEBUG: Task 720 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 721 has memoization hash ebddc8d5e319f43b771b24d9f8a2e150
parsl.dataflow.memoization INFO: Task 721 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 600
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb62cb70-ad0e-4e82-9ee6-1f6b44cb779f_characterizeImage_947422_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb62cb70-ad0e-4e82-9ee6-1f6b44cb779f_characterizeImage_947422_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/991b0057-89e2-4f20-a4d8-e51a998457ec_isr_947422_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/991b0057-89e2-4f20-a4d8-e51a998457ec_isr_947422_48.stderr
parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8272d6f7898574a7eb1b0bc1f00e89eb with result from task 729
parsl.dataflow.dflow DEBUG: Task 729 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 068f9d77aa25aed51298b022d993c704
parsl.dataflow.memoization INFO: Task 730 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 601
parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/95e7f283-485f-4c71-abd1-1f11854b5806_characterizeImage_947420_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/95e7f283-485f-4c71-abd1-1f11854b5806_characterizeImage_947420_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4b69b6a4-25f2-482c-a405-e1042b7e2ff1_isr_947420_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/4b69b6a4-25f2-482c-a405-e1042b7e2ff1_isr_947420_19.stderr
parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dff0795067d75f0c4b19b28517e8abd9 with result from task 714
parsl.dataflow.dflow DEBUG: Task 714 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 715 has memoization hash c3f9369c7b3ff175fe093c2a913e4b5f
parsl.dataflow.memoization INFO: Task 715 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 602
parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9d5a5885-339d-47d2-8d41-5040bd39ec46_characterizeImage_947523_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9d5a5885-339d-47d2-8d41-5040bd39ec46_characterizeImage_947523_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/d7d27b5a-9626-4782-a8c9-9670aa65748e_isr_947523_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/d7d27b5a-9626-4782-a8c9-9670aa65748e_isr_947523_62.stderr
parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d53d5b37f8af6d6d0b45896b26703770 with result from task 711
parsl.dataflow.dflow DEBUG: Task 711 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 97d9cee766808b8e006ab023c748e720
parsl.dataflow.memoization INFO: Task 712 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 603
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1e2fd78-0200-47ad-b600-b389e6732d81_characterizeImage_947523_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1e2fd78-0200-47ad-b600-b389e6732d81_characterizeImage_947523_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/71a06537-8313-4031-9a29-510210d54f0f_isr_947523_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/71a06537-8313-4031-9a29-510210d54f0f_isr_947523_44.stderr
parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 469c787d240ba34f56dbfa2a6bbf3ca8 with result from task 585
parsl.dataflow.dflow DEBUG: Task 585 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 71211f679387cf4f64cd0bb3e515a50f
parsl.dataflow.memoization INFO: Task 586 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 604
parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7215405d-a113-460c-8bea-53ce36b7cba3_characterizeImage_947523_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7215405d-a113-460c-8bea-53ce36b7cba3_characterizeImage_947523_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/ebdd6fda-6535-49cb-93c8-e8946bf05ee0_isr_947523_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/ebdd6fda-6535-49cb-93c8-e8946bf05ee0_isr_947523_1.stderr
parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f93a72dad14874801ac3ad73cdb72bf with result from task 567
parsl.dataflow.dflow DEBUG: Task 567 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 79f8fcdd82a4d34c9918e2a16749dd1a
parsl.dataflow.memoization INFO: Task 568 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 605
parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/60239d29-7502-4917-b05b-3bea7460301c_characterizeImage_947420_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/60239d29-7502-4917-b05b-3bea7460301c_characterizeImage_947420_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9f8579cc-848f-4068-86ab-44c8dae9c955_isr_947420_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/9f8579cc-848f-4068-86ab-44c8dae9c955_isr_947420_12.stderr
parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f79043ec174c2db6584e4de6fe9e9ad5 with result from task 582
parsl.dataflow.dflow DEBUG: Task 582 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 54d250247014943d330af91ff728503f
parsl.dataflow.memoization INFO: Task 583 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 606
parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/61fdccf7-12fb-46f5-8116-2563a4e06eea_characterizeImage_947524_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/61fdccf7-12fb-46f5-8116-2563a4e06eea_characterizeImage_947524_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9005f966-5a16-4228-9c11-9624e504fbd2_isr_947524_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9005f966-5a16-4228-9c11-9624e504fbd2_isr_947524_30.stderr
parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43b4c38753307c9ee76971e004b98322 with result from task 573
parsl.dataflow.dflow DEBUG: Task 573 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 34728f90581cf55b1d4961bc2913bc7d
parsl.dataflow.memoization INFO: Task 574 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 607
parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/371ef057-2bf2-4296-bc8a-6cb1ce7b5ce0_characterizeImage_947524_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/371ef057-2bf2-4296-bc8a-6cb1ce7b5ce0_characterizeImage_947524_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/269d1423-0ac3-4181-a4e3-9a36909ae0d9_isr_947524_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/269d1423-0ac3-4181-a4e3-9a36909ae0d9_isr_947524_29.stderr
parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9238ff43e76bab4a22b147aa28882094 with result from task 471
parsl.dataflow.dflow DEBUG: Task 471 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 6c9d83cc0e374bdeb6605e77e80d63f8
parsl.dataflow.memoization INFO: Task 472 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 608
parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/138da8fc-69c9-4539-b110-5c5247326e98_characterizeImage_947524_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/138da8fc-69c9-4539-b110-5c5247326e98_characterizeImage_947524_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4f262436-003b-4e1a-af9a-5a95fc51799d_isr_947524_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4f262436-003b-4e1a-af9a-5a95fc51799d_isr_947524_27.stderr
parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7754fc09147f5d55bacdcf1fc311a088 with result from task 726
parsl.dataflow.dflow DEBUG: Task 726 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 6580879b48fe8c04c914e6ad601ac836
parsl.dataflow.memoization INFO: Task 727 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 609
parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d63cd63a-912f-4385-92c4-f5a510c9f88a_characterizeImage_947418_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d63cd63a-912f-4385-92c4-f5a510c9f88a_characterizeImage_947418_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/94737fc6-01d8-4d65-9038-1aecd31305fa_isr_947418_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/94737fc6-01d8-4d65-9038-1aecd31305fa_isr_947418_48.stderr
parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d71120519c48a465075c48f213239962 with result from task 732
parsl.dataflow.dflow DEBUG: Task 732 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 5f5c01d363ba84ad7ee208c79c7e3ecc
parsl.dataflow.memoization INFO: Task 733 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:51 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 610
parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8a8da536-b8d5-43e5-81ae-5c3f867dd53c_characterizeImage_947524_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8a8da536-b8d5-43e5-81ae-5c3f867dd53c_characterizeImage_947524_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/156cbabe-87c7-4aa0-bb65-66f0eef27c64_isr_947524_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/156cbabe-87c7-4aa0-bb65-66f0eef27c64_isr_947524_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:51 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:51 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:51 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:51 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:52 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02f99a6dc215bd7f9bd29929c716f897 with result from task 501
parsl.dataflow.dflow DEBUG: Task 501 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 1660065923c7ec4e583bbde972be9de0
parsl.dataflow.memoization INFO: Task 502 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 611
parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/d8bc5c8a-367d-470d-844d-d0d2c28ae4f8_characterizeImage_947528_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/d8bc5c8a-367d-470d-844d-d0d2c28ae4f8_characterizeImage_947528_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/44839bb4-e012-475a-897e-874592835039_isr_947528_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/44839bb4-e012-475a-897e-874592835039_isr_947528_1.stderr
parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0112de60fc483748bfdc06373cb5295c with result from task 492
parsl.dataflow.dflow DEBUG: Task 492 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 13c9cfe299b2d2f727b9834c8b668973
parsl.dataflow.memoization INFO: Task 493 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 612
parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ba2e67e2-acbf-458b-b979-bde2a101eada_characterizeImage_947418_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ba2e67e2-acbf-458b-b979-bde2a101eada_characterizeImage_947418_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/53cbe677-7f32-455c-a258-7b35ac69e014_isr_947418_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/53cbe677-7f32-455c-a258-7b35ac69e014_isr_947418_34.stderr
parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ba62f37dcd9112eac469ac6b00eef6e with result from task 495
parsl.dataflow.dflow DEBUG: Task 495 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 496 has memoization hash bdd46f582065b04fe9a64f7c86c99bf2
parsl.dataflow.memoization INFO: Task 496 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 613
parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/feb4bdae-825f-4944-9072-d8a5ed54dc67_characterizeImage_947523_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/feb4bdae-825f-4944-9072-d8a5ed54dc67_characterizeImage_947523_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/998f989c-adcd-4a07-bba4-12acda02faa9_isr_947523_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/998f989c-adcd-4a07-bba4-12acda02faa9_isr_947523_52.stderr
parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93f5e3e2b55e51b27fd0f04bcac5642b with result from task 510
parsl.dataflow.dflow DEBUG: Task 510 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 905d106e83ee1fb755efb3dde71b5c7c
parsl.dataflow.memoization INFO: Task 511 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 614
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/9d80d1a0-8fe8-43e4-b394-1765d68c0265_characterizeImage_947528_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/9d80d1a0-8fe8-43e4-b394-1765d68c0265_characterizeImage_947528_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c69df377-7c21-4737-8abe-ff0d7093716a_isr_947528_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c69df377-7c21-4737-8abe-ff0d7093716a_isr_947528_7.stderr
parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6ab2cb129902d1957b622d5ef9ffa88 with result from task 513
parsl.dataflow.dflow DEBUG: Task 513 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 2d49d00ca7244ef1094a60834f54f211
parsl.dataflow.memoization INFO: Task 514 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 615
parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/db93699a-7a00-486e-af52-10cfa04e899b_characterizeImage_947526_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/db93699a-7a00-486e-af52-10cfa04e899b_characterizeImage_947526_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/61923aa0-ab2e-4512-ac53-a5dbf51982ba_isr_947526_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/61923aa0-ab2e-4512-ac53-a5dbf51982ba_isr_947526_49.stderr
parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46819fe6c1157c3c90f4f81fe71a0db8 with result from task 507
parsl.dataflow.dflow DEBUG: Task 507 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 7dd66ca082839f04a1b7fd5bc7a590e4
parsl.dataflow.memoization INFO: Task 508 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 616
parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/9e04b372-407d-417d-8711-2e4d7912590e_characterizeImage_947526_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/9e04b372-407d-417d-8711-2e4d7912590e_characterizeImage_947526_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e56fb22a-f7f2-4c59-8d83-31d682f2e852_isr_947526_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/e56fb22a-f7f2-4c59-8d83-31d682f2e852_isr_947526_37.stderr
parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 462d4b6bb2390c7259fc36890242454a with result from task 684
parsl.dataflow.dflow DEBUG: Task 684 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 351ff48daf46919e3e633065cddc92e8
parsl.dataflow.memoization INFO: Task 685 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 617
parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/141242a6-4cd0-4157-93d6-8eed67302f9c_characterizeImage_947418_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/141242a6-4cd0-4157-93d6-8eed67302f9c_characterizeImage_947418_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b737d956-790c-40ef-8a5b-97d73f16486f_isr_947418_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b737d956-790c-40ef-8a5b-97d73f16486f_isr_947418_31.stderr
parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd64cab7dc4196a448e8eb7032049128 with result from task 504
parsl.dataflow.dflow DEBUG: Task 504 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 505 has memoization hash a1b8cd575c1af764417f7697d5cb41d3
parsl.dataflow.memoization INFO: Task 505 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 618
parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9cca75ac-c5cc-4208-ac32-9f7b7b6584eb_characterizeImage_947524_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9cca75ac-c5cc-4208-ac32-9f7b7b6584eb_characterizeImage_947524_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75584bd1-9c2e-41ac-a541-7355daecbdd4_isr_947524_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/75584bd1-9c2e-41ac-a541-7355daecbdd4_isr_947524_13.stderr
parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6e230a925055b5b74afa18f04947f61 with result from task 498
parsl.dataflow.dflow DEBUG: Task 498 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 550e3c3176644d271013ce3b865d95e9
parsl.dataflow.memoization INFO: Task 499 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 619
parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a9c06fef-a94b-4e67-8f79-5f91fcd505fd_characterizeImage_947523_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a9c06fef-a94b-4e67-8f79-5f91fcd505fd_characterizeImage_947523_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/c9963555-260d-47b6-930e-c7b49302ed2a_isr_947523_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/c9963555-260d-47b6-930e-c7b49302ed2a_isr_947523_50.stderr
parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7d5e305971201bb32be65fa002f22e7 with result from task 666
parsl.dataflow.dflow DEBUG: Task 666 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 667 has memoization hash bd1220829bf7493b8e39b16975e2c6a2
parsl.dataflow.memoization INFO: Task 667 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 620
parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0db3de36-e22e-40e1-8637-5ae1e1242a00_characterizeImage_947420_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0db3de36-e22e-40e1-8637-5ae1e1242a00_characterizeImage_947420_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/51d42e7d-a1e7-4581-b813-4f180f5d0388_isr_947420_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/51d42e7d-a1e7-4581-b813-4f180f5d0388_isr_947420_28.stderr
parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fbe8754d269e911417aeb9f242ced1ef with result from task 681
parsl.dataflow.dflow DEBUG: Task 681 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 117474f87c599340f246d2658905a74d
parsl.dataflow.memoization INFO: Task 682 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 621
parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fb08e93c-c322-4999-bb5c-e1ba2e0fcd22_characterizeImage_947528_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fb08e93c-c322-4999-bb5c-e1ba2e0fcd22_characterizeImage_947528_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bcf96450-37b5-46d6-befd-74ee72cf7924_isr_947528_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bcf96450-37b5-46d6-befd-74ee72cf7924_isr_947528_62.stderr
parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc23e9f510af184192dd0e56e3e8238f with result from task 669
parsl.dataflow.dflow DEBUG: Task 669 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 7a5bcabccf633bde7a25b4b391e55a9b
parsl.dataflow.memoization INFO: Task 670 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 622
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ec3d92b3-bb82-4efb-868b-cbaa5960f596_characterizeImage_947420_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ec3d92b3-bb82-4efb-868b-cbaa5960f596_characterizeImage_947420_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/28a028dc-01ec-4f2d-9b90-475bd8eb563f_isr_947420_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/28a028dc-01ec-4f2d-9b90-475bd8eb563f_isr_947420_35.stderr
parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f56c955680c2e28d78f6e6fe7624e608 with result from task 678
parsl.dataflow.dflow DEBUG: Task 678 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 15100f163a195fffadc096dae8eb75bd
parsl.dataflow.memoization INFO: Task 679 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 623
parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f4c7c2f5-5ae3-40b1-946d-889e2a8dc698_characterizeImage_947422_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f4c7c2f5-5ae3-40b1-946d-889e2a8dc698_characterizeImage_947422_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c4dfdd93-19e8-4448-995c-11a369974d8c_isr_947422_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c4dfdd93-19e8-4448-995c-11a369974d8c_isr_947422_55.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 352 try 0 failed
parsl.dataflow.dflow INFO: Task 352 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 4cfec2f9edbbcea86f056e813319bd42
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 1 launched on executor multi with executor id 624
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stderr
parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5edd5f969fa23780d3737d17e39d1d20 with result from task 675
parsl.dataflow.dflow DEBUG: Task 675 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 676 has memoization hash dff5cf7fa99bcadc5bee4b1bfd7bbf73
parsl.dataflow.memoization INFO: Task 676 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 625
parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cea84278-0920-4507-ab1c-d71d2be72094_characterizeImage_947418_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cea84278-0920-4507-ab1c-d71d2be72094_characterizeImage_947418_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d0f28290-0b57-4f13-b110-82f5fd551725_isr_947418_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d0f28290-0b57-4f13-b110-82f5fd551725_isr_947418_55.stderr
parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1c247fcb130d7b46d96dda5f509e4df with result from task 663
parsl.dataflow.dflow DEBUG: Task 663 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 664 has memoization hash eac5d0679f44d796630590cd316615ee
parsl.dataflow.memoization INFO: Task 664 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 626
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e473f9fd-3fc4-4d5d-b10a-486b93ade134_characterizeImage_947422_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e473f9fd-3fc4-4d5d-b10a-486b93ade134_characterizeImage_947422_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/51371269-7373-482b-acab-f14804720e35_isr_947422_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/51371269-7373-482b-acab-f14804720e35_isr_947422_31.stderr
parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b95bce87394a896fbc19df12d9d230dd with result from task 672
parsl.dataflow.dflow DEBUG: Task 672 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 3d71b23fdfbc92b2f032d5be7e21ec70
parsl.dataflow.memoization INFO: Task 673 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 627
parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/12d5d1ba-8d7a-440a-9305-11b46d7aa5ce_characterizeImage_947422_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/12d5d1ba-8d7a-440a-9305-11b46d7aa5ce_characterizeImage_947422_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b38a1018-d7c6-41d6-8c85-96f6d469410c_isr_947422_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b38a1018-d7c6-41d6-8c85-96f6d469410c_isr_947422_29.stderr
parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b49a51a135ddbc85a80c87ef1c4cde54 with result from task 807
parsl.dataflow.dflow DEBUG: Task 807 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 808 has memoization hash e7ec5fde666151ac18315aef04148327
parsl.dataflow.memoization INFO: Task 808 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 628
parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/994c376f-f467-4d93-b895-fbd14bf1a35e_characterizeImage_947523_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/994c376f-f467-4d93-b895-fbd14bf1a35e_characterizeImage_947523_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fdd87d7b-e4f3-4341-a456-75fe39ced94d_isr_947523_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fdd87d7b-e4f3-4341-a456-75fe39ced94d_isr_947523_33.stderr
parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51ce13e4101f7d26ad8ea8cad87622e4 with result from task 819
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:52 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow DEBUG: Task 819 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 42d2f384e86c8425b11915cafb852a29
parsl.dataflow.memoization INFO: Task 820 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d084a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 629
parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/119ec7b9-5a77-4f17-8812-0cdd0a513aad_characterizeImage_947526_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/119ec7b9-5a77-4f17-8812-0cdd0a513aad_characterizeImage_947526_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4898456f-9696-4ade-9ab1-fd0f2f735d0d_isr_947526_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4898456f-9696-4ade-9ab1-fd0f2f735d0d_isr_947526_15.stderr
parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa7447229b08ff35f2299b5cdabf58e with result from task 753
parsl.dataflow.dflow DEBUG: Task 753 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 461da71da082bcf18d7ebb2ad3f850d9
parsl.dataflow.memoization INFO: Task 754 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 630
parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8659ec58-f07a-4513-85b3-d3bf00214833_characterizeImage_947523_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8659ec58-f07a-4513-85b3-d3bf00214833_characterizeImage_947523_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7836fd93-b6cf-438d-a30e-bbf41e6d355d_isr_947523_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7836fd93-b6cf-438d-a30e-bbf41e6d355d_isr_947523_17.stderr
parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 94bd4c0551a756cc70f3e904f09770e0 with result from task 738
parsl.dataflow.dflow DEBUG: Task 738 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 0984d7e60638aead998944ca96bf7a59
parsl.dataflow.memoization INFO: Task 739 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 631
parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/be692336-b29d-4f73-b386-822817191a3b_characterizeImage_947422_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/be692336-b29d-4f73-b386-822817191a3b_characterizeImage_947422_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0f762b41-cc37-46f3-b657-add23d7c99fe_isr_947422_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0f762b41-cc37-46f3-b657-add23d7c99fe_isr_947422_28.stderr
parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66db025262529447ddf6cf0e1999c5ef with result from task 735
parsl.dataflow.dflow DEBUG: Task 735 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 73e955dbea052a72dc20580b6d70a9bd
parsl.dataflow.memoization INFO: Task 736 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 632
parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f60f078e-5783-478e-b7d6-49a39d04c10e_characterizeImage_947528_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f60f078e-5783-478e-b7d6-49a39d04c10e_characterizeImage_947528_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/79891680-d9ea-4e28-870c-6375b4cfc02f_isr_947528_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/79891680-d9ea-4e28-870c-6375b4cfc02f_isr_947528_27.stderr
parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1bac6392a4e780bc4b21970c51f72474 with result from task 756
parsl.dataflow.dflow DEBUG: Task 756 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 757 has memoization hash e62ec62b9ec0e9bdedba2130bcd57284
parsl.dataflow.memoization INFO: Task 757 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 633
parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/54db3523-7e7e-40d8-95cf-b805cf8e235f_characterizeImage_947523_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/54db3523-7e7e-40d8-95cf-b805cf8e235f_characterizeImage_947523_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b9e0f4f8-e3da-487f-98ce-604b0f8f4dc5_isr_947523_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b9e0f4f8-e3da-487f-98ce-604b0f8f4dc5_isr_947523_40.stderr
parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f290585e26af439c091d0d3445ccd6ce with result from task 744
parsl.dataflow.dflow DEBUG: Task 744 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 44678e3ad0a8ffd2230c5c2472eb6ca4
parsl.dataflow.memoization INFO: Task 745 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 634
parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/daf5e1b3-c944-4897-b39e-4df43b29002f_characterizeImage_947524_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/daf5e1b3-c944-4897-b39e-4df43b29002f_characterizeImage_947524_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0a05a638-ae4c-4dc5-b606-b6499d9bc393_isr_947524_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0a05a638-ae4c-4dc5-b606-b6499d9bc393_isr_947524_45.stderr
parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f06c221734fd9e6e45aa2379a0232e2b with result from task 750
parsl.dataflow.dflow DEBUG: Task 750 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 751 has memoization hash be9e3163f8457a48c4bae57977f365fa
parsl.dataflow.memoization INFO: Task 751 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 635
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/205e6d1f-31ff-41e0-90b7-a54c2ee5d559_characterizeImage_947422_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/205e6d1f-31ff-41e0-90b7-a54c2ee5d559_characterizeImage_947422_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/180fc813-90c2-41ec-ac13-4d3ae16aa25f_isr_947422_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/180fc813-90c2-41ec-ac13-4d3ae16aa25f_isr_947422_22.stderr
parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6830a3e598c47285601b90abed4eb0c with result from task 615
parsl.dataflow.dflow DEBUG: Task 615 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 7e4e85ee64681f47d8cf34d09c2c2476
parsl.dataflow.memoization INFO: Task 616 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 636
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/58b6e822-5a51-4ff8-af2d-980872eab0fd_characterizeImage_947526_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/58b6e822-5a51-4ff8-af2d-980872eab0fd_characterizeImage_947526_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5347f695-2223-43d2-a7ce-3565cb6a6faa_isr_947526_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5347f695-2223-43d2-a7ce-3565cb6a6faa_isr_947526_9.stderr
parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d186cb806b26bb8ae3e7c931bbb78f18 with result from task 657
parsl.dataflow.dflow DEBUG: Task 657 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 658 has memoization hash d7c10d8c7f94b0112f5f3d03a70c019e
parsl.dataflow.memoization INFO: Task 658 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 637
parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/85cf8048-f25e-4974-bbfc-b754b0eb9e3c_characterizeImage_947526_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/85cf8048-f25e-4974-bbfc-b754b0eb9e3c_characterizeImage_947526_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/144becd8-5739-4163-a497-a9d15135e45d_isr_947526_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/144becd8-5739-4163-a497-a9d15135e45d_isr_947526_56.stderr
parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e363c86babcc9a6c89a9323236dc60f with result from task 645
parsl.dataflow.dflow DEBUG: Task 645 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 03c2e6ec5be85eccd11351c9da58da78
parsl.dataflow.memoization INFO: Task 646 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 638
parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2287180e-0227-4a47-8eca-b88583de519e_characterizeImage_947422_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2287180e-0227-4a47-8eca-b88583de519e_characterizeImage_947422_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2f878301-e9a1-4b8f-b0cf-280903cc72c8_isr_947422_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2f878301-e9a1-4b8f-b0cf-280903cc72c8_isr_947422_56.stderr
parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0ee03dc495efb6854d61fc793e9f177 with result from task 660
parsl.dataflow.dflow DEBUG: Task 660 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 661 has memoization hash 21da732757a7b391975bb42e29e6d740
parsl.dataflow.memoization INFO: Task 661 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 639
parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/753e4f36-5215-4490-bb7c-6456e22e73f8_characterizeImage_947526_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/753e4f36-5215-4490-bb7c-6456e22e73f8_characterizeImage_947526_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6ff96fad-8cf2-4f1f-a226-d4b3d29c567a_isr_947526_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6ff96fad-8cf2-4f1f-a226-d4b3d29c567a_isr_947526_8.stderr
parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 71aa318501ad6d0b33f5b1d7071e4dc9 with result from task 639
parsl.dataflow.dflow DEBUG: Task 639 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 0d927821c0e72d930dd0a0c149a8f59d
parsl.dataflow.memoization INFO: Task 640 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 640
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/77031e25-b203-48a0-846d-e28aa37b026b_characterizeImage_947528_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/77031e25-b203-48a0-846d-e28aa37b026b_characterizeImage_947528_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2e162512-ff1b-4a2a-95d9-1c4d680e5da3_isr_947528_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2e162512-ff1b-4a2a-95d9-1c4d680e5da3_isr_947528_33.stderr
parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 61edcc07be2d4913d5c8b7f68dc00c14 with result from task 612
parsl.dataflow.dflow DEBUG: Task 612 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 1ae48b8dc36a58cc23b8c2fc899d7d83
parsl.dataflow.memoization INFO: Task 613 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 641
parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/33306b88-d18f-4587-b457-b7f2a9185bbc_characterizeImage_947422_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/33306b88-d18f-4587-b457-b7f2a9185bbc_characterizeImage_947422_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/853956f0-e029-44af-b79b-939a5ef4a871_isr_947422_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/853956f0-e029-44af-b79b-939a5ef4a871_isr_947422_39.stderr
parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2c187bd5e9d1ffa9006a658f5d6d771 with result from task 609
parsl.dataflow.dflow DEBUG: Task 609 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 610 has memoization hash c736ecad6e55a626b18122b4d8061d04
parsl.dataflow.memoization INFO: Task 610 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 642
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/980ac996-1a2f-4304-adcc-88ffd72a0b75_characterizeImage_947526_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/980ac996-1a2f-4304-adcc-88ffd72a0b75_characterizeImage_947526_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/12a48e3a-00e8-4dd6-ab91-bd2b43c77f51_isr_947526_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/12a48e3a-00e8-4dd6-ab91-bd2b43c77f51_isr_947526_62.stderr
parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb3c75f8b9bd8edb742d188ea7e36520 with result from task 600
parsl.dataflow.dflow DEBUG: Task 600 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 20e797b4dd30bc86aafba4b640a09978
parsl.dataflow.memoization INFO: Task 601 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 643
parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ea5fd53e-4c7c-4683-838c-9fdc0bee02a6_characterizeImage_947526_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ea5fd53e-4c7c-4683-838c-9fdc0bee02a6_characterizeImage_947526_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/812a58cb-ad4e-4bda-81ab-0c837769aed9_isr_947526_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/812a58cb-ad4e-4bda-81ab-0c837769aed9_isr_947526_27.stderr
parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c34b1fa7cd96dc93854b76d814559779 with result from task 606
parsl.dataflow.dflow DEBUG: Task 606 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 13f16a203b04f3df42bd42eeecf47264
parsl.dataflow.memoization INFO: Task 607 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 644
parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7a53135e-bc27-4380-8058-0f156dcb8d35_characterizeImage_947418_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7a53135e-bc27-4380-8058-0f156dcb8d35_characterizeImage_947418_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3008513e-abd5-469a-ab8d-4ec4b855d921_isr_947418_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3008513e-abd5-469a-ab8d-4ec4b855d921_isr_947418_19.stderr
parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 45e688c1338c2c94f2f87db5f7a97e42 with result from task 633
parsl.dataflow.dflow DEBUG: Task 633 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 46fdb20551de014880f21721c811c6ce
parsl.dataflow.memoization INFO: Task 634 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 645
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c7364642-399e-49e0-8efe-d8256646f47c_characterizeImage_947418_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c7364642-399e-49e0-8efe-d8256646f47c_characterizeImage_947418_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5ad49684-06d9-4128-a7cc-a92bb3fa458c_isr_947418_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5ad49684-06d9-4128-a7cc-a92bb3fa458c_isr_947418_45.stderr
parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e483685af79aa25ef57d7126641bd9c6 with result from task 627
parsl.dataflow.dflow DEBUG: Task 627 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 628 has memoization hash cc424e1dd99f251a18d0898e8aff1761
parsl.dataflow.memoization INFO: Task 628 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 646
parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/48685365-85fd-497b-85f9-dd7516aa6fe0_characterizeImage_947422_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/48685365-85fd-497b-85f9-dd7516aa6fe0_characterizeImage_947422_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c31eb02e-4092-49fa-b2b9-1a2d49b75ec0_isr_947422_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/c31eb02e-4092-49fa-b2b9-1a2d49b75ec0_isr_947422_49.stderr
parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20af12715548692cbcaf92432da1e319 with result from task 654
parsl.dataflow.dflow DEBUG: Task 654 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 26fbdaf691b05d66b42ee78021a29118
parsl.dataflow.memoization INFO: Task 655 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 647
parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ca1ed468-d589-4013-ae44-4ae7c85afa6a_characterizeImage_947523_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ca1ed468-d589-4013-ae44-4ae7c85afa6a_characterizeImage_947523_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cb8b6332-7802-46ec-ab7b-a4e58035d7e4_isr_947523_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/cb8b6332-7802-46ec-ab7b-a4e58035d7e4_isr_947523_4.stderr
parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e218044316eac1815c6d0591c41bd03 with result from task 651
parsl.dataflow.dflow DEBUG: Task 651 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 652 has memoization hash a7315e77a2a4947307e61bf2053bdf2a
parsl.dataflow.memoization INFO: Task 652 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 648
parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f2f62467-87c0-48a9-864c-0e04a4aa5b2c_characterizeImage_947526_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f2f62467-87c0-48a9-864c-0e04a4aa5b2c_characterizeImage_947526_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6165cb49-1271-46b0-a7e6-27c4443cb9c5_isr_947526_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6165cb49-1271-46b0-a7e6-27c4443cb9c5_isr_947526_26.stderr
parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9bb92606b6887b09d2e665db44b7bb7b with result from task 594
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:52 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow DEBUG: Task 594 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 595 has memoization hash ff7508b26ccee4df03d5caa6240e1fd2
parsl.dataflow.memoization INFO: Task 595 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 649
parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/430c2d69-5983-4670-b26d-bf4013902e42_characterizeImage_947422_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/430c2d69-5983-4670-b26d-bf4013902e42_characterizeImage_947422_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b1053073-5cf5-4b60-a08f-61415e2c3278_isr_947422_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b1053073-5cf5-4b60-a08f-61415e2c3278_isr_947422_53.stderr
parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 641490bb1c03fcaea2ac8536929b5670 with result from task 591
parsl.dataflow.dflow DEBUG: Task 591 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 592 has memoization hash a4d197240efd7cb7828362fb7ae5422f
parsl.dataflow.memoization INFO: Task 592 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 650
parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/91bcd2a8-7cbf-4298-bea5-4404c1bd5074_characterizeImage_947422_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/91bcd2a8-7cbf-4298-bea5-4404c1bd5074_characterizeImage_947422_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/428cc318-bcd9-44bb-bef7-799f0fde279d_isr_947422_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/428cc318-bcd9-44bb-bef7-799f0fde279d_isr_947422_13.stderr
parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59fff7bfb0ef407fc8e67627251fb897 with result from task 624
parsl.dataflow.dflow DEBUG: Task 624 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 5ebfc3542540835b982448e077e3ae58
parsl.dataflow.memoization INFO: Task 625 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 651
parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/14e3fd95-1388-48c0-b8d8-430e411162ab_characterizeImage_947420_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/14e3fd95-1388-48c0-b8d8-430e411162ab_characterizeImage_947420_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d7dc86f2-8e0f-4179-851b-1c8b222f22b8_isr_947420_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d7dc86f2-8e0f-4179-851b-1c8b222f22b8_isr_947420_20.stderr
parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c9e29fc65659433d84f06297178505e with result from task 1047
parsl.dataflow.dflow DEBUG: Task 1047 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash fc8ecf38cee0dece62918cc0863cf862
parsl.dataflow.memoization INFO: Task 1048 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 652
parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057064e4-780b-4927-ab2d-74252e043754_characterizeImage_947420_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057064e4-780b-4927-ab2d-74252e043754_characterizeImage_947420_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/01faa5ef-1df7-4fa7-a0e9-fa16e78a7c63_isr_947420_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/01faa5ef-1df7-4fa7-a0e9-fa16e78a7c63_isr_947420_57.stderr
parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11fc42d3309f102a515642e1264ffec6 with result from task 741
parsl.dataflow.dflow DEBUG: Task 741 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 16eb1e5119d25e127a5fdac05745ed47
parsl.dataflow.memoization INFO: Task 742 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 653
parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/866191ff-0656-43ba-853f-74227b19ceb4_characterizeImage_947524_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/866191ff-0656-43ba-853f-74227b19ceb4_characterizeImage_947524_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/412090d0-18be-46df-a163-8232d842da9e_isr_947524_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/412090d0-18be-46df-a163-8232d842da9e_isr_947524_7.stderr
parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e56e75b4195ba15bf6e34367a530349 with result from task 816
parsl.dataflow.dflow DEBUG: Task 816 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 1b2556275f85c8fa7704e4fd2ef1a8ba
parsl.dataflow.memoization INFO: Task 817 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 654
parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7ab48534-8386-4da4-ac70-894611b3f81c_characterizeImage_947524_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7ab48534-8386-4da4-ac70-894611b3f81c_characterizeImage_947524_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0297da2-2c23-4093-911e-187021eb8f2b_isr_947524_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0297da2-2c23-4093-911e-187021eb8f2b_isr_947524_48.stderr
parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 183d4b0ab5c745d74039f3a7c565cf77 with result from task 747
parsl.dataflow.dflow DEBUG: Task 747 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 23ce941d6bf4f4c6aedf5fcecfb91f1b
parsl.dataflow.memoization INFO: Task 748 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 655
parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/e08b4bad-7b4c-4d66-a053-e02972feabfe_characterizeImage_947523_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/e08b4bad-7b4c-4d66-a053-e02972feabfe_characterizeImage_947523_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/301c801c-8c29-4ee9-bf3d-a6f2dab45a70_isr_947523_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/301c801c-8c29-4ee9-bf3d-a6f2dab45a70_isr_947523_38.stderr
parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85505869a04d0cc4c94f28c4fc9ee210 with result from task 762
parsl.dataflow.dflow DEBUG: Task 762 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 028caa3c15625281839eb75dfecf3514
parsl.dataflow.memoization INFO: Task 763 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 656
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6c115c9b-d7bf-4d94-8dab-a708c81941bc_characterizeImage_947528_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6c115c9b-d7bf-4d94-8dab-a708c81941bc_characterizeImage_947528_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f161eb3c-3d1a-45f0-bb3c-d8be45f14274_isr_947528_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f161eb3c-3d1a-45f0-bb3c-d8be45f14274_isr_947528_60.stderr
parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9317b4a700b664fa22c61b9edf9864da with result from task 771
parsl.dataflow.dflow DEBUG: Task 771 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 55a25a65ab935ea2138243d7a2be8fad
parsl.dataflow.memoization INFO: Task 772 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 657
parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be049c80-c8f5-46fd-abbe-ea0f01320356_characterizeImage_947420_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be049c80-c8f5-46fd-abbe-ea0f01320356_characterizeImage_947420_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/b465e888-84ec-481a-a021-1cf38cd4d17b_isr_947420_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/b465e888-84ec-481a-a021-1cf38cd4d17b_isr_947420_39.stderr
parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3307fd58b7b0574b6fe90424302f294d with result from task 780
parsl.dataflow.dflow DEBUG: Task 780 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 781 has memoization hash bb596d8fa8d74d35cb1e2b280ca390e9
parsl.dataflow.memoization INFO: Task 781 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 658
parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6dc89df7-22a1-43a8-b44f-a1d88965a934_characterizeImage_947422_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6dc89df7-22a1-43a8-b44f-a1d88965a934_characterizeImage_947422_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/134b8e20-ae21-4805-8772-cac90daa9276_isr_947422_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/134b8e20-ae21-4805-8772-cac90daa9276_isr_947422_44.stderr
parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52bc2ad880aa2ce8fe63bccd9e0b5a0b with result from task 774
parsl.dataflow.dflow DEBUG: Task 774 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 8b5f814cf11dd56c8a76876616fd461f
parsl.dataflow.memoization INFO: Task 775 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 659
parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5d2dec1f-4ba6-4878-8813-fe61947c835f_characterizeImage_947528_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5d2dec1f-4ba6-4878-8813-fe61947c835f_characterizeImage_947528_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/476299b0-c08c-464c-b2d7-e569ce4bc021_isr_947528_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/476299b0-c08c-464c-b2d7-e569ce4bc021_isr_947528_58.stderr
parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e6ed2156808fb2ef452cfdc5b51bfdb with result from task 765
parsl.dataflow.dflow DEBUG: Task 765 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 90e9f23a19ed26d4e28ba673e02a6f44
parsl.dataflow.memoization INFO: Task 766 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 660
parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/abad35b8-a904-436a-bd7f-a27a4fa90057_characterizeImage_947528_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/abad35b8-a904-436a-bd7f-a27a4fa90057_characterizeImage_947528_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/607ebb04-74fc-4f24-869c-be5bfa2c5915_isr_947528_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/607ebb04-74fc-4f24-869c-be5bfa2c5915_isr_947528_14.stderr
parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8bba5a123f942d3e6501a49fff7bbf22 with result from task 768
parsl.dataflow.dflow DEBUG: Task 768 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 0b8d58def7b739e980ea6850425b3b26
parsl.dataflow.memoization INFO: Task 769 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 661
parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8c7c9be0-1485-48b5-a91c-5c6b8c19570a_characterizeImage_947524_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8c7c9be0-1485-48b5-a91c-5c6b8c19570a_characterizeImage_947524_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/29b568c4-11c4-49ba-8654-2115fbefa49a_isr_947524_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/29b568c4-11c4-49ba-8654-2115fbefa49a_isr_947524_42.stderr
parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7141b367461812a0b12b8be4acd09ab2 with result from task 603
parsl.dataflow.dflow DEBUG: Task 603 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 604 has memoization hash 85a5fde0ec29070cb72a2266f22086e9
parsl.dataflow.memoization INFO: Task 604 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 662
parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7230a5cd-fee0-4552-8ddc-905fc96f67a7_characterizeImage_947524_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7230a5cd-fee0-4552-8ddc-905fc96f67a7_characterizeImage_947524_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/888a9dc1-b2e8-4597-b25f-61c46853e0fd_isr_947524_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/888a9dc1-b2e8-4597-b25f-61c46853e0fd_isr_947524_62.stderr
parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4a4649c5df9edc2056229eb92aca335 with result from task 777
parsl.dataflow.dflow DEBUG: Task 777 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 778 has memoization hash fe6c89ed3bcd58f8247a712df1067e17
parsl.dataflow.memoization INFO: Task 778 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 663
parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4455ea7c-b94a-492c-bb91-afb4597743d2_characterizeImage_947523_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4455ea7c-b94a-492c-bb91-afb4597743d2_characterizeImage_947523_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2bfab70a-6240-4540-8571-76ec8a20288c_isr_947523_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/2bfab70a-6240-4540-8571-76ec8a20288c_isr_947523_61.stderr
parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c5f52a4ebca10b548f112cbb2158ec0 with result from task 480
parsl.dataflow.dflow DEBUG: Task 480 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 481 has memoization hash db1983a908bf28b81bbe6147a4c07ef1
parsl.dataflow.memoization INFO: Task 481 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 664
parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8a1436da-1f38-488a-895c-dfdf4ee89ec9_characterizeImage_947418_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8a1436da-1f38-488a-895c-dfdf4ee89ec9_characterizeImage_947418_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb3a7c28-211c-4016-af61-911ea573e9fa_isr_947418_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/bb3a7c28-211c-4016-af61-911ea573e9fa_isr_947418_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:53 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00451eaa80cf740c8baa0972b965a70e with result from task 1059
parsl.dataflow.dflow DEBUG: Task 1059 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 5b147cf768fb831e25dd7897d538763c
parsl.dataflow.memoization INFO: Task 1060 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 665
parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fc1fa03b-6b24-4369-a628-9e99a47ab8cb_characterizeImage_947526_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fc1fa03b-6b24-4369-a628-9e99a47ab8cb_characterizeImage_947526_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6f824794-fa1c-41ec-a186-2ce2265aea78_isr_947526_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6f824794-fa1c-41ec-a186-2ce2265aea78_isr_947526_39.stderr
parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e92b607b40de9335795a61050a1ba347 with result from task 597
parsl.dataflow.dflow DEBUG: Task 597 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 598 has memoization hash f2a577cf072892beb94f9175a54045d9
parsl.dataflow.memoization INFO: Task 598 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 666
parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f2a3d5b6-1c64-4a78-82a3-6b6acbb72258_characterizeImage_947422_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f2a3d5b6-1c64-4a78-82a3-6b6acbb72258_characterizeImage_947422_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2dce7b54-10d2-43a9-8370-cb4c0b695061_isr_947422_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2dce7b54-10d2-43a9-8370-cb4c0b695061_isr_947422_38.stderr
parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 992b2bdaa8f7bfe584a22c73e8dd92bb with result from task 636
parsl.dataflow.dflow DEBUG: Task 636 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 637 has memoization hash f47a53f776e488aedb732f793abe3405
parsl.dataflow.memoization INFO: Task 637 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 667
parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e5ca0c2f-9fff-4ebd-b3db-0a6a3a4d01e0_characterizeImage_947420_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e5ca0c2f-9fff-4ebd-b3db-0a6a3a4d01e0_characterizeImage_947420_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3c4768e4-6523-4b21-9419-68290dac75ce_isr_947420_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3c4768e4-6523-4b21-9419-68290dac75ce_isr_947420_41.stderr
parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52c7c654421713f5442a573a07dbcd0e with result from task 618
parsl.dataflow.dflow DEBUG: Task 618 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 619 has memoization hash bdf72acb9fd71f5d729bd99dd5dda967
parsl.dataflow.memoization INFO: Task 619 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 668
parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/774ee501-80d2-4a26-a178-588780a94fad_characterizeImage_947526_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/774ee501-80d2-4a26-a178-588780a94fad_characterizeImage_947526_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0a97e0e4-4525-4341-802c-6343a452b462_isr_947526_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0a97e0e4-4525-4341-802c-6343a452b462_isr_947526_60.stderr
parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 867ab45e5e196fbd8bf7336a3f5a8ff7 with result from task 1041
parsl.dataflow.dflow DEBUG: Task 1041 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 47506b7d9bf4116965b89bd51520ea17
parsl.dataflow.memoization INFO: Task 1042 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 669
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ba68bde8-5dd2-4c13-b4c3-0fa09ec9e512_characterizeImage_947526_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ba68bde8-5dd2-4c13-b4c3-0fa09ec9e512_characterizeImage_947526_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bcfb6b0b-cd6d-4e58-b399-7266b633e673_isr_947526_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bcfb6b0b-cd6d-4e58-b399-7266b633e673_isr_947526_17.stderr
parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2895b64c04911025e79133d52ce910b with result from task 621
parsl.dataflow.dflow DEBUG: Task 621 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 32987da6a0d573e147b3b1750200665d
parsl.dataflow.memoization INFO: Task 622 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 670
parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fb4d9c2e-e8df-4160-8c5a-98668802a21c_characterizeImage_947523_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fb4d9c2e-e8df-4160-8c5a-98668802a21c_characterizeImage_947523_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/03c347ad-d0e2-4d3c-ac18-af3c8e612493_isr_947523_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/03c347ad-d0e2-4d3c-ac18-af3c8e612493_isr_947523_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:53 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6017e5db79d732eddd5b3c3e7411e002 with result from task 1056
parsl.dataflow.dflow DEBUG: Task 1056 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash 391a21e3f50bde5397e56ee31d717367
parsl.dataflow.memoization INFO: Task 1057 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 671
parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/659e3651-f548-46d5-85df-39fc19d50ba5_characterizeImage_947528_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/659e3651-f548-46d5-85df-39fc19d50ba5_characterizeImage_947528_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bf077fa1-f099-46e9-b5a1-24f6ecf8e268_isr_947528_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/bf077fa1-f099-46e9-b5a1-24f6ecf8e268_isr_947528_23.stderr
parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d781dd616df96e1878b42205eab3138 with result from task 1044
parsl.dataflow.dflow DEBUG: Task 1044 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash b192ef54f393445ec3a7327718e948ac
parsl.dataflow.memoization INFO: Task 1045 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 672
parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/86663901-8359-428a-9af5-0b1af26b7256_characterizeImage_947422_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/86663901-8359-428a-9af5-0b1af26b7256_characterizeImage_947422_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/fa97f8bd-b59e-4781-8120-1643d6611d51_isr_947422_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/fa97f8bd-b59e-4781-8120-1643d6611d51_isr_947422_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:54 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a2e26729177f4d370e4b242cb8477fa with result from task 1050
parsl.dataflow.dflow DEBUG: Task 1050 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash d1035dec94ecf856bbdf73fa63ccfeee
parsl.dataflow.memoization INFO: Task 1051 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 673
parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/63c30d0c-a705-4e69-ba3c-215ff2d982ef_characterizeImage_947528_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/63c30d0c-a705-4e69-ba3c-215ff2d982ef_characterizeImage_947528_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9b36a470-55a3-44bc-952c-fa64970b73a4_isr_947528_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9b36a470-55a3-44bc-952c-fa64970b73a4_isr_947528_49.stderr
parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 147c8365a470fe301345f7aa767d0331 with result from task 810
parsl.dataflow.dflow DEBUG: Task 810 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 811 has memoization hash d2086e30432abd52d3d4fd7120000d20
parsl.dataflow.memoization INFO: Task 811 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 674
parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/39330eef-8750-4e53-9987-6a58ffe5564d_characterizeImage_947418_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/39330eef-8750-4e53-9987-6a58ffe5564d_characterizeImage_947418_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3a6ba6e9-0453-4291-8285-f771668f5f23_isr_947418_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3a6ba6e9-0453-4291-8285-f771668f5f23_isr_947418_35.stderr
parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0217a49fcb7143a36ab62b2b4ae147d8 with result from task 1089
parsl.dataflow.dflow DEBUG: Task 1089 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 120d471a4d52bfb9ee1384be220b41b5
parsl.dataflow.memoization INFO: Task 1090 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 675
parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fd4df93-474d-498d-9eba-b56ebc1adeb5_characterizeImage_947524_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fd4df93-474d-498d-9eba-b56ebc1adeb5_characterizeImage_947524_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/85cb5bb9-7b86-4098-9f58-f6a151a2d5f5_isr_947524_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/85cb5bb9-7b86-4098-9f58-f6a151a2d5f5_isr_947524_56.stderr
parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0d8bab40f2708ed582a8ebb95aa504f with result from task 1080
parsl.dataflow.dflow DEBUG: Task 1080 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash 726ad2d1579fab5e12cdb3078c40e0c4
parsl.dataflow.memoization INFO: Task 1081 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 676
parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9e09a2cc-96e3-4ccd-8362-00eda259d74a_characterizeImage_947524_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9e09a2cc-96e3-4ccd-8362-00eda259d74a_characterizeImage_947524_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2ec66d38-7a23-4fe7-b20b-55c54ad09b3b_isr_947524_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2ec66d38-7a23-4fe7-b20b-55c54ad09b3b_isr_947524_47.stderr
parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f6508dc80de3c1c967e84cc6d044d04 with result from task 1062
parsl.dataflow.dflow DEBUG: Task 1062 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 75f371bfb84ab0898bdb6da9f06c67d8
parsl.dataflow.memoization INFO: Task 1063 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 677
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/dc371f14-2c74-437c-aa36-eb6badd2a9f3_characterizeImage_947528_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/dc371f14-2c74-437c-aa36-eb6badd2a9f3_characterizeImage_947528_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/d9da1282-7531-4bd6-bc5d-496888e1bb23_isr_947528_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/d9da1282-7531-4bd6-bc5d-496888e1bb23_isr_947528_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:54 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23445edb42e3cac16f6df240c85494f5 with result from task 642
parsl.dataflow.dflow DEBUG: Task 642 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 6cd19485f6c50a425cfd1a6217a5dcd1
parsl.dataflow.memoization INFO: Task 643 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 678
parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ae18e70c-28f0-4702-9ca4-21558ab40af9_characterizeImage_947524_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ae18e70c-28f0-4702-9ca4-21558ab40af9_characterizeImage_947524_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce178b88-649e-4714-9b45-4e1f34fc164e_isr_947524_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce178b88-649e-4714-9b45-4e1f34fc164e_isr_947524_49.stderr
parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d5c1a8a827f6075df1771964383413c with result from task 825
parsl.dataflow.dflow DEBUG: Task 825 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 08698e97273c58875b187f7347172945
parsl.dataflow.memoization INFO: Task 826 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 679
parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/78a60f3b-a900-49eb-958d-43eafffb8900_characterizeImage_947528_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/78a60f3b-a900-49eb-958d-43eafffb8900_characterizeImage_947528_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/df7e5251-fd39-45e9-899a-4a29c6d924aa_isr_947528_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/df7e5251-fd39-45e9-899a-4a29c6d924aa_isr_947528_56.stderr
parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecb89e14a70d60c41fc5a58a69c8ff94 with result from task 822
parsl.dataflow.dflow DEBUG: Task 822 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 0a5b5c575f137d0b32e42ce46b3e655e
parsl.dataflow.memoization INFO: Task 823 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d087c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 680
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d1cf777-3f7b-4ebf-b70c-181b84e6e156_characterizeImage_947528_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d1cf777-3f7b-4ebf-b70c-181b84e6e156_characterizeImage_947528_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/851612ad-54c3-4d74-a02f-e2513bc5484e_isr_947528_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/851612ad-54c3-4d74-a02f-e2513bc5484e_isr_947528_37.stderr
parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 073667e9cb134ca0a144aeec912cdaf1 with result from task 1083
parsl.dataflow.dflow DEBUG: Task 1083 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash e75c8d717664da82441a13ad0d1c3be4
parsl.dataflow.memoization INFO: Task 1084 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 681
parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/37eef965-1ef3-47dd-b485-1f82383d4ed0_characterizeImage_947528_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/37eef965-1ef3-47dd-b485-1f82383d4ed0_characterizeImage_947528_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/ff65a48c-8765-46a4-a19d-fcca10e7e555_isr_947528_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/ff65a48c-8765-46a4-a19d-fcca10e7e555_isr_947528_24.stderr
parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3948c4410ec363f3ce182dce4b69162c with result from task 1074
parsl.dataflow.dflow DEBUG: Task 1074 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash c976d23c9038ed9f51781cc2f6acdf60
parsl.dataflow.memoization INFO: Task 1075 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 682
parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/85722506-226c-48c8-a828-d31adcbf2f0c_characterizeImage_947422_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/85722506-226c-48c8-a828-d31adcbf2f0c_characterizeImage_947422_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a5e1d854-49ff-4ea8-a3c3-980d435604df_isr_947422_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a5e1d854-49ff-4ea8-a3c3-980d435604df_isr_947422_11.stderr
parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22bf10b362a2ae595208c9f459ac83ca with result from task 1086
parsl.dataflow.dflow DEBUG: Task 1086 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash 3d7a5691512dde946d0867937ef649ea
parsl.dataflow.memoization INFO: Task 1087 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 683
parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/049b854d-7cc1-4c4b-b6fe-c8c5b4335793_characterizeImage_947528_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/049b854d-7cc1-4c4b-b6fe-c8c5b4335793_characterizeImage_947528_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/511843d9-ed88-4821-94cf-8ec8ae0219ba_isr_947528_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/511843d9-ed88-4821-94cf-8ec8ae0219ba_isr_947528_9.stderr
parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d90397a681a3dfb0a2ab6a82b9d0c659 with result from task 1053
parsl.dataflow.dflow DEBUG: Task 1053 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash d5e6885770f5cd9f0e9f4f02907a9e36
parsl.dataflow.memoization INFO: Task 1054 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 684
parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3390dacc-fdb8-4d01-8827-f10a14488192_characterizeImage_947523_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3390dacc-fdb8-4d01-8827-f10a14488192_characterizeImage_947523_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5518c18a-e637-4d3c-b468-b01a43faafcb_isr_947523_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5518c18a-e637-4d3c-b468-b01a43faafcb_isr_947523_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:54 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0ddfa8f0191cb8f0f9aa11d9246fee3 with result from task 648
parsl.dataflow.dflow DEBUG: Task 648 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 224e52a84abdbf5688956a6437c648f8
parsl.dataflow.memoization INFO: Task 649 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 685
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d8cb596b-f33d-482a-97f2-28c8f379292f_characterizeImage_947422_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d8cb596b-f33d-482a-97f2-28c8f379292f_characterizeImage_947422_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/47acf2f8-8840-4dde-a4f6-0dec752c4059_isr_947422_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/47acf2f8-8840-4dde-a4f6-0dec752c4059_isr_947422_16.stderr
parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e896ce2a2c102a93a842f862736e24d7 with result from task 1095
parsl.dataflow.dflow DEBUG: Task 1095 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash be3c4348063610ffabdad6fdf54dfe8d
parsl.dataflow.memoization INFO: Task 1096 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 686
parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fb3fb52-e7a0-41f4-bba1-6405f49923af_characterizeImage_947524_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fb3fb52-e7a0-41f4-bba1-6405f49923af_characterizeImage_947524_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/053f1a53-fbc6-4f05-9978-050ea50222ba_isr_947524_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/053f1a53-fbc6-4f05-9978-050ea50222ba_isr_947524_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:55 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 085505e044c7577a28594d468217ac36 with result from task 759
parsl.dataflow.dflow DEBUG: Task 759 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 79bd184581f4918019b8a428d9efd47f
parsl.dataflow.memoization INFO: Task 760 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 687
parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9e2f4441-75a6-4e32-a25a-00d4f9aa5e37_characterizeImage_947420_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9e2f4441-75a6-4e32-a25a-00d4f9aa5e37_characterizeImage_947420_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bea57d7-e3dc-424c-acb1-24877ca19f76_isr_947420_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/7bea57d7-e3dc-424c-acb1-24877ca19f76_isr_947420_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:55 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e1dca6cb617849199991e375dcddde0d with result from task 1077
parsl.dataflow.dflow DEBUG: Task 1077 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash fa9c0cda603f449e5d83b160d388bf29
parsl.dataflow.memoization INFO: Task 1078 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 688
parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af9584ac-a574-4b62-b5b5-9385e381746d_characterizeImage_947528_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af9584ac-a574-4b62-b5b5-9385e381746d_characterizeImage_947528_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/877b7d36-95c9-4d12-9221-1cf7875591c2_isr_947528_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/877b7d36-95c9-4d12-9221-1cf7875591c2_isr_947528_12.stderr
parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43105f9778ac527c55d6f38fb194de93 with result from task 630
parsl.dataflow.dflow DEBUG: Task 630 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 631 has memoization hash fea46b37a28392823df0afb216a23ae3
parsl.dataflow.memoization INFO: Task 631 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 689
parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4c6089a6-e545-453c-b735-410e59c61d30_characterizeImage_947523_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4c6089a6-e545-453c-b735-410e59c61d30_characterizeImage_947523_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0287b4ba-df83-40e9-b633-4fee07200799_isr_947523_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0287b4ba-df83-40e9-b633-4fee07200799_isr_947523_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:56 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:57 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 132fc0086eb1668ff3bd305ca4d84e42 with result from task 891
parsl.dataflow.dflow DEBUG: Task 891 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 203582603327a11ddb55d0d2ed2addf7
parsl.dataflow.memoization INFO: Task 892 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 690
parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4fb07e3c-4a52-4150-82a2-d1b301399ec7_characterizeImage_947422_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4fb07e3c-4a52-4150-82a2-d1b301399ec7_characterizeImage_947422_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/d285b61f-5ec0-416e-83c2-48eaf75989f7_isr_947422_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/d285b61f-5ec0-416e-83c2-48eaf75989f7_isr_947422_14.stderr
parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecb810772c1e5e267cbd808a2c75f50b with result from task 888
parsl.dataflow.dflow DEBUG: Task 888 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 6b3e8df186814cd6d7eb17c762a2417e
parsl.dataflow.memoization INFO: Task 889 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 691
parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/0a13a308-5862-4faa-994b-096b32579127_characterizeImage_947526_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/0a13a308-5862-4faa-994b-096b32579127_characterizeImage_947526_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4fce6d0d-ff21-4ce0-859b-470b5bea8e82_isr_947526_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/4fce6d0d-ff21-4ce0-859b-470b5bea8e82_isr_947526_33.stderr
parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df031e179c1b97a4bb422ec705eeb0c6 with result from task 897
parsl.dataflow.dflow DEBUG: Task 897 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 9db612024c015dcb56eade0701f9db06
parsl.dataflow.memoization INFO: Task 898 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 692
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/867b1965-a335-4f22-a3f5-292835011cb0_isr_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/867b1965-a335-4f22-a3f5-292835011cb0_isr_947526_2.stderr
parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c24c6b0573011462849cafa781c6eb3 with result from task 900
parsl.dataflow.dflow DEBUG: Task 900 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 901 has memoization hash c1db15c9b01152fcd8ac5235ea79aeef
parsl.dataflow.memoization INFO: Task 901 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 693
parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e8570474-5be8-4913-919e-6ae08b807ab4_characterizeImage_947422_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e8570474-5be8-4913-919e-6ae08b807ab4_characterizeImage_947422_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a8af7d8-0b72-4f2c-b9f0-5c0a7643dc72_isr_947422_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a8af7d8-0b72-4f2c-b9f0-5c0a7643dc72_isr_947422_58.stderr
parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21149f887c7ea003c72d83ad16536ba8 with result from task 909
parsl.dataflow.dflow DEBUG: Task 909 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 910 has memoization hash f884b0e9f7b2414e10a4fb88db8fa343
parsl.dataflow.memoization INFO: Task 910 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 694
parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1dcc9b22-d583-48f4-aabb-247f6b6ae438_characterizeImage_947422_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1dcc9b22-d583-48f4-aabb-247f6b6ae438_characterizeImage_947422_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a42a57c1-44d9-4218-b188-d1fbecce4586_isr_947422_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/a42a57c1-44d9-4218-b188-d1fbecce4586_isr_947422_1.stderr
parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9eb30509d7fce294e09d2ebcf2fd4ab with result from task 903
parsl.dataflow.dflow DEBUG: Task 903 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 904 has memoization hash e4e2ab6ca7b5b162a8523d5e5d5a3948
parsl.dataflow.memoization INFO: Task 904 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 695
parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7682c06b-84a1-42c0-b4aa-1ba6fce09ba5_characterizeImage_947422_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7682c06b-84a1-42c0-b4aa-1ba6fce09ba5_characterizeImage_947422_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2529e872-deae-4647-bc18-dbc2736a23fa_isr_947422_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2529e872-deae-4647-bc18-dbc2736a23fa_isr_947422_41.stderr
parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0a444f6ef756fb71d33e0e5773f2c65 with result from task 906
parsl.dataflow.dflow DEBUG: Task 906 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 5519ed4dd97ccb375ebf1f1019093fcb
parsl.dataflow.memoization INFO: Task 907 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 696
parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9dd1babb-c795-476d-89a4-8b118dd4d222_characterizeImage_947523_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9dd1babb-c795-476d-89a4-8b118dd4d222_characterizeImage_947523_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5695aa7f-c6df-4f95-8fab-feb095a579a5_isr_947523_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/5695aa7f-c6df-4f95-8fab-feb095a579a5_isr_947523_15.stderr
parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58d818f1f2bba9a81176f6c571438594 with result from task 894
parsl.dataflow.dflow DEBUG: Task 894 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 59f536a0ba90775ff70eaaa723bd53f4
parsl.dataflow.memoization INFO: Task 895 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 697
parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7a64a0e4-cc08-4cf3-af93-03378a54bde4_characterizeImage_947528_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7a64a0e4-cc08-4cf3-af93-03378a54bde4_characterizeImage_947528_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11603f74-603f-46ef-99f2-35cffd3d5e50_isr_947528_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11603f74-603f-46ef-99f2-35cffd3d5e50_isr_947528_4.stderr
parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18f9b2e614f019f6e2f3440f253f85d4 with result from task 1092
parsl.dataflow.dflow DEBUG: Task 1092 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash c7092425514555cd0f644f3c7392d0ed
parsl.dataflow.memoization INFO: Task 1093 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 698
parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/adcc61b8-24de-48e8-99ea-6ddc0bbd1967_characterizeImage_947420_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/adcc61b8-24de-48e8-99ea-6ddc0bbd1967_characterizeImage_947420_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d18447df-c3a7-49f5-a2ce-76ed08306afd_isr_947420_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/d18447df-c3a7-49f5-a2ce-76ed08306afd_isr_947420_54.stderr
parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccf17d9b7f19f3a3ff99c1f9e58d7eaf with result from task 843
parsl.dataflow.dflow DEBUG: Task 843 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 3de045d8cb19e70877801bd3a90d3e23
parsl.dataflow.memoization INFO: Task 844 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 699
parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/702851f0-5475-467f-b99f-e156ccc829ec_characterizeImage_947523_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/702851f0-5475-467f-b99f-e156ccc829ec_characterizeImage_947523_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b0678ad6-eef2-490d-9aad-05078fa32c13_isr_947523_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b0678ad6-eef2-490d-9aad-05078fa32c13_isr_947523_5.stderr
parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e83c67758c1623afaa4ec59ae2517bbc with result from task 870
parsl.dataflow.dflow DEBUG: Task 870 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 871 has memoization hash f2f482c92e152d2cc289ad5adc21c730
parsl.dataflow.memoization INFO: Task 871 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 700
parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9afdde3c-37eb-48d2-9016-959261e95f19_characterizeImage_947420_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9afdde3c-37eb-48d2-9016-959261e95f19_characterizeImage_947420_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/29dddbfb-b367-4307-a69b-c8d4f7425d24_isr_947420_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/29dddbfb-b367-4307-a69b-c8d4f7425d24_isr_947420_32.stderr
parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b42c1fadab3e4487157d10723dd0729a with result from task 876
parsl.dataflow.dflow DEBUG: Task 876 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 16dc9a0de465a05712a4ca0e53e1d4c8
parsl.dataflow.memoization INFO: Task 877 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 701
parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b941cd9-91b6-4bf6-a2f8-1a4e99a1d7f4_characterizeImage_947418_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b941cd9-91b6-4bf6-a2f8-1a4e99a1d7f4_characterizeImage_947418_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c5a2d528-fbf9-4701-8c5c-b06e386b5d4b_isr_947418_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c5a2d528-fbf9-4701-8c5c-b06e386b5d4b_isr_947418_33.stderr
parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c9dbdf4037024ac6f1ace4e4de8b421 with result from task 849
parsl.dataflow.dflow DEBUG: Task 849 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 9765c2004f57d99add9a65ecef8a9461
parsl.dataflow.memoization INFO: Task 850 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 702
parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/adfe1bb8-609c-43c4-af28-9bc4d58af67f_characterizeImage_947523_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/adfe1bb8-609c-43c4-af28-9bc4d58af67f_characterizeImage_947523_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1264088f-ec8a-4b1e-bc77-20874890dbc3_isr_947523_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/1264088f-ec8a-4b1e-bc77-20874890dbc3_isr_947523_41.stderr
parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00ff97e7173ef6e13b4d4949000be61c with result from task 837
parsl.dataflow.dflow DEBUG: Task 837 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 72eab86fb06a9928680384f38ec552f6
parsl.dataflow.memoization INFO: Task 838 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 703
parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/77d4440d-ed41-4625-9263-621bd0f45f0a_characterizeImage_947420_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/77d4440d-ed41-4625-9263-621bd0f45f0a_characterizeImage_947420_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8e735054-31cf-4dec-875d-b827213f64fd_isr_947420_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8e735054-31cf-4dec-875d-b827213f64fd_isr_947420_31.stderr
parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae89a3ab6dd04c5d8bec9c9ec6a1433a with result from task 846
parsl.dataflow.dflow DEBUG: Task 846 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 1140f05a98bcbf106825ef9df45b32c4
parsl.dataflow.memoization INFO: Task 847 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 704
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8e1f7857-822e-4f41-b8e5-99be0a8902a4_characterizeImage_947524_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8e1f7857-822e-4f41-b8e5-99be0a8902a4_characterizeImage_947524_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ff138c26-ec66-4994-88ba-25285c3bc7c9_isr_947524_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ff138c26-ec66-4994-88ba-25285c3bc7c9_isr_947524_4.stderr
parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07be38c7454ba8e212ad1996608c407e with result from task 852
parsl.dataflow.dflow DEBUG: Task 852 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 19466c7321a03ec50517cd21585f78ab
parsl.dataflow.memoization INFO: Task 853 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 705
parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4ec997f2-ef1e-42ab-840a-f3106bfcde70_characterizeImage_947524_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4ec997f2-ef1e-42ab-840a-f3106bfcde70_characterizeImage_947524_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6025c777-f1d6-46f9-a822-38858f9d9cfe_isr_947524_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6025c777-f1d6-46f9-a822-38858f9d9cfe_isr_947524_51.stderr
parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c339cd816e4540624e228ef75dcd6dc0 with result from task 867
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:57 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow DEBUG: Task 867 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 868 has memoization hash dadaa3c5ea687321a57de92afd52a274
parsl.dataflow.memoization INFO: Task 868 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 706
parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b67fd7f-485b-4384-884d-e4fb1fd4e39c_characterizeImage_947526_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b67fd7f-485b-4384-884d-e4fb1fd4e39c_characterizeImage_947526_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/354f8e7e-cece-4ad7-8022-50e41e90be85_isr_947526_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/354f8e7e-cece-4ad7-8022-50e41e90be85_isr_947526_10.stderr
parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdf96d875f7cccad9b76252f025bf30 with result from task 873
parsl.dataflow.dflow DEBUG: Task 873 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 874 has memoization hash f568fc097a413e22adf3eba3239a3999
parsl.dataflow.memoization INFO: Task 874 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 707
parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cc139760-7b9a-427a-b6c6-a722ea5cf2f9_characterizeImage_947528_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cc139760-7b9a-427a-b6c6-a722ea5cf2f9_characterizeImage_947528_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f92ba361-b3bd-4c65-acbf-f49f86a8f210_isr_947528_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/f92ba361-b3bd-4c65-acbf-f49f86a8f210_isr_947528_11.stderr
parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae0ee6d31e547824875015dc0a2834e5 with result from task 855
parsl.dataflow.dflow DEBUG: Task 855 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 8374bd19cb934ac43d925b00dd4c921f
parsl.dataflow.memoization INFO: Task 856 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 708
parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8b409464-20b2-4eb8-a38c-fe6d7136f59b_characterizeImage_947523_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8b409464-20b2-4eb8-a38c-fe6d7136f59b_characterizeImage_947523_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/849467f6-7cf7-4751-b373-c0ce8abbda38_isr_947523_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/849467f6-7cf7-4751-b373-c0ce8abbda38_isr_947523_53.stderr
parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31e68444690e5e6f03b01148e745fcee with result from task 858
parsl.dataflow.dflow DEBUG: Task 858 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 812d18db21f2f8732847befbc29b3082
parsl.dataflow.memoization INFO: Task 859 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 709
parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/398e9a70-353e-4379-8654-b8dcd5a3e6c5_characterizeImage_947422_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/398e9a70-353e-4379-8654-b8dcd5a3e6c5_characterizeImage_947422_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a174c0e-0d5d-45cd-971d-4405713ea66c_isr_947422_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5a174c0e-0d5d-45cd-971d-4405713ea66c_isr_947422_51.stderr
parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 75cfe17e12ddacce3461e08ba9bc9fd6 with result from task 864
parsl.dataflow.dflow DEBUG: Task 864 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 33b2663c71ed9cb5a8784b3f102a13d2
parsl.dataflow.memoization INFO: Task 865 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 710
parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/a23af7c2-c24e-4df9-83f3-4a35ecff1ece_characterizeImage_947526_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/a23af7c2-c24e-4df9-83f3-4a35ecff1ece_characterizeImage_947526_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8ad6c19b-49cb-4562-883b-2c44040f2a0c_isr_947526_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8ad6c19b-49cb-4562-883b-2c44040f2a0c_isr_947526_16.stderr
parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a36aafc010a96881b887744563eb0d22 with result from task 861
parsl.dataflow.dflow DEBUG: Task 861 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 9dc61d4b89caea9a42648f4a04cf3429
parsl.dataflow.memoization INFO: Task 862 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 711
parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af64cb5f-7c4e-42fa-8edb-bb6d090bfe73_characterizeImage_947420_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af64cb5f-7c4e-42fa-8edb-bb6d090bfe73_characterizeImage_947420_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c6bd6c40-eb1b-4d6e-914f-71cd49afc967_isr_947420_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c6bd6c40-eb1b-4d6e-914f-71cd49afc967_isr_947420_18.stderr
parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1e126313d3c1aa379d2414c6739bdd5 with result from task 549
parsl.dataflow.dflow DEBUG: Task 549 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 3c04b51adb4b98cc04c2cf1f09a677b6
parsl.dataflow.memoization INFO: Task 550 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 712
parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d52a49d3-b7d0-4686-8e45-adc19f6fde9e_characterizeImage_947418_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d52a49d3-b7d0-4686-8e45-adc19f6fde9e_characterizeImage_947418_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fb7cf08c-cdb8-461a-ac29-97afa6e7fb16_isr_947418_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fb7cf08c-cdb8-461a-ac29-97afa6e7fb16_isr_947418_56.stderr
parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4662a96a3f583d3bc02bf29e9a8726fd with result from task 522
parsl.dataflow.dflow DEBUG: Task 522 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 62d80ac1480c9d7bbbf18b3ec6ecdd8b
parsl.dataflow.memoization INFO: Task 523 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 713
parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/98aef122-daa5-4bf0-a3d4-0fc33681dd7c_characterizeImage_947422_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/98aef122-daa5-4bf0-a3d4-0fc33681dd7c_characterizeImage_947422_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b4a52109-2dbb-4f10-b9ea-e3c08f4a930c_isr_947422_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/b4a52109-2dbb-4f10-b9ea-e3c08f4a930c_isr_947422_12.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f89d32ec42c5545546d249cd8d076c7b with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash bc49e8b69ea9accdab3c7e369e4e5efa
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 714
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/63030db4-caaa-4b82-8981-0880ed3973fe_characterizeImage_947524_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/63030db4-caaa-4b82-8981-0880ed3973fe_characterizeImage_947524_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ca29301-2b94-44b6-96dc-2b007a7d9ea7_isr_947524_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/4ca29301-2b94-44b6-96dc-2b007a7d9ea7_isr_947524_24.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8612627cfca317a9cd6aaa40837fa7f3 with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash b49253f40f78dd8587e7b9fa12851ede
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 715
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c0df162f-8f7d-4732-8e10-58a5cbbde875_characterizeImage_947418_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c0df162f-8f7d-4732-8e10-58a5cbbde875_characterizeImage_947418_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0ce01b95-ee20-4d6d-9905-9e32006a78cf_isr_947418_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0ce01b95-ee20-4d6d-9905-9e32006a78cf_isr_947418_49.stderr
parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f278eb2b5ea2bd726ad015e1f68f2c6 with result from task 534
parsl.dataflow.dflow DEBUG: Task 534 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 12b25f8a2b82fcf5a59bad56afa3b0cb
parsl.dataflow.memoization INFO: Task 535 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 716
parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/7a5a6818-80d5-4e51-b0f2-b85af1725bd0_characterizeImage_947526_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/7a5a6818-80d5-4e51-b0f2-b85af1725bd0_characterizeImage_947526_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c06eecdd-32ee-4de6-9127-c1c7222a5ca0_isr_947526_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c06eecdd-32ee-4de6-9127-c1c7222a5ca0_isr_947526_32.stderr
parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eede28835262f068cf1f0727326f9c65 with result from task 555
parsl.dataflow.dflow DEBUG: Task 555 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 556 has memoization hash f25c36a3802e8c14f2c35ed61bfd755b
parsl.dataflow.memoization INFO: Task 556 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 717
parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ae3b2afa-9e3e-4997-b57a-38033d50cd65_characterizeImage_947418_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ae3b2afa-9e3e-4997-b57a-38033d50cd65_characterizeImage_947418_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0a400bf3-7abc-4f6f-ac42-158d3eb89113_isr_947418_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0a400bf3-7abc-4f6f-ac42-158d3eb89113_isr_947418_5.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50c5aecd50d4595e241960b4110b1d9a with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 623cc275c57ff512fdc2fb588f16e794
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e967a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 718
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fa36c989-89c7-4d73-a7e8-624f35727681_isr_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/fa36c989-89c7-4d73-a7e8-624f35727681_isr_947418_2.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb79b34f5c7e4b6008969c898a17d6d9 with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 077570f6bbb13416bbffd9ec421a86b3
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 719
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ce6f248e-9f5f-4cd0-8a1c-7b742e477932_characterizeImage_947418_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ce6f248e-9f5f-4cd0-8a1c-7b742e477932_characterizeImage_947418_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3537d5af-bca9-4ff7-9a21-3d43c0a823a9_isr_947418_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/3537d5af-bca9-4ff7-9a21-3d43c0a823a9_isr_947418_14.stderr
parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c73fca0fe0c224304baa4c85abbf9609 with result from task 198
parsl.dataflow.dflow DEBUG: Task 198 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 53a9c98cd0b186c1bcec4461302dff5e
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 720
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1f1786ee-a348-46c7-8db0-4ffe50ab5818_characterizeImage_947418_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1f1786ee-a348-46c7-8db0-4ffe50ab5818_characterizeImage_947418_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f8bf7cd9-7f9a-4b7b-b92e-6fcab7b6a897_isr_947418_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/f8bf7cd9-7f9a-4b7b-b92e-6fcab7b6a897_isr_947418_30.stderr
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d71743060ce278f294eec4270977274 with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash b8b3cdf0fe6043f2197ffdf973f54ec6
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 721
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/15d0aefb-d75a-49b4-a810-8fcf2897b6f2_characterizeImage_947524_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/15d0aefb-d75a-49b4-a810-8fcf2897b6f2_characterizeImage_947524_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/82958b06-7bd7-4184-abef-f366f4db9810_isr_947524_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/82958b06-7bd7-4184-abef-f366f4db9810_isr_947524_41.stderr
parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d6fb6a7744d11a893ce32714e03bb189 with result from task 192
parsl.dataflow.dflow DEBUG: Task 192 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 193 has memoization hash a0a676506ab7fc2178914fd5a1243966
parsl.dataflow.memoization INFO: Task 193 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 722
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fcdfec6d-ed95-4b4f-8c62-0577fba8f901_characterizeImage_947526_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fcdfec6d-ed95-4b4f-8c62-0577fba8f901_characterizeImage_947526_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/dc4bb19b-5c4b-4743-93a2-9d9227e52a2a_isr_947526_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/dc4bb19b-5c4b-4743-93a2-9d9227e52a2a_isr_947526_30.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1418e9a4a86d54c11635725bbe378596 with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash f832bf4ccf7b75300bec7a7dd62f3737
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 723
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/49e60a25-8fe4-4f6c-9bec-c9c6455cfd75_characterizeImage_947528_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/49e60a25-8fe4-4f6c-9bec-c9c6455cfd75_characterizeImage_947528_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/67e78330-8374-422d-8db5-ce9c4d1850a1_isr_947528_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/67e78330-8374-422d-8db5-ce9c4d1850a1_isr_947528_25.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba5eb6d71ce733842e089cc0af97739b with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 23f5dbcdcef49be29ada7c75315af027
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 724
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ec696ee2-d0ae-488b-8d00-302bc06926b6_characterizeImage_947523_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ec696ee2-d0ae-488b-8d00-302bc06926b6_characterizeImage_947523_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/33ded963-9b80-4b5d-b779-890eb0602868_isr_947523_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/33ded963-9b80-4b5d-b779-890eb0602868_isr_947523_46.stderr
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c592f4531543c03634fc3d58ffbc71e with result from task 153
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:57 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 72c5e640fc9990491cee0c5ee4caf204
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 725
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ce17b12d-3d6d-4c5e-b7f2-95ce4c80a0d2_characterizeImage_947526_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ce17b12d-3d6d-4c5e-b7f2-95ce4c80a0d2_characterizeImage_947526_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c1f7bcbc-51e4-4a80-ac64-034c56468f41_isr_947526_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c1f7bcbc-51e4-4a80-ac64-034c56468f41_isr_947526_22.stderr
parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99c1934268d27b7ba72dbee15852d960 with result from task 186
parsl.dataflow.dflow DEBUG: Task 186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 2801331b19ee2857a0f5302a4b9d9973
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 726
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/464ab7bd-71d0-4f64-8084-70cdc13f2f69_characterizeImage_947420_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/464ab7bd-71d0-4f64-8084-70cdc13f2f69_characterizeImage_947420_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f977efd3-7dba-46bf-adb5-f29057d1ae35_isr_947420_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f977efd3-7dba-46bf-adb5-f29057d1ae35_isr_947420_4.stderr
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0d0a7f1aeb68ae875bf1a88970a653e with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 145008bc2011f379b9128ef1ea1421a1
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 727
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2c9a6f8f-8fbf-4f82-b091-56677244b580_characterizeImage_947418_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2c9a6f8f-8fbf-4f82-b091-56677244b580_characterizeImage_947418_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/eb2cfc6e-de34-4389-9823-bd24711f2751_isr_947418_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/eb2cfc6e-de34-4389-9823-bd24711f2751_isr_947418_4.stderr
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 790f0de8391890fdadb50bd3a583fc67 with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 6eca07188d8751c916a532f01d0289a2
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 728
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/0b19434a-e963-4bc1-9e4e-02507c278d0b_characterizeImage_947523_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/0b19434a-e963-4bc1-9e4e-02507c278d0b_characterizeImage_947523_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8ac755b3-0a1c-4a1b-8246-f2d1e8f43277_isr_947523_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8ac755b3-0a1c-4a1b-8246-f2d1e8f43277_isr_947523_21.stderr
parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e278472c4d0be944f03b96b96e0b5acf with result from task 552
parsl.dataflow.dflow DEBUG: Task 552 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 8c2ededd99d6d8fd90be60326e2e9e35
parsl.dataflow.memoization INFO: Task 553 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 729
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/e53968f1-e465-4248-9042-c85e0109fa00_isr_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/e53968f1-e465-4248-9042-c85e0109fa00_isr_947524_2.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8bb5cc00cc1530284ec973aa619875 with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash ac6518c821cd4bda4059a47d6946b71c
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 730
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b9c1ecd9-ce10-478d-a033-0c9c455f353c_characterizeImage_947526_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b9c1ecd9-ce10-478d-a033-0c9c455f353c_characterizeImage_947526_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d4e2d5a3-bd32-445d-8208-6f196a89bfca_isr_947526_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d4e2d5a3-bd32-445d-8208-6f196a89bfca_isr_947526_40.stderr
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc3c1fbd2e188e777f8dbdb007ef4937 with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash fb55a2addb1f49eff44c83900bcebb95
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 731
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6cf5b653-a70d-4f3b-a80b-c492fb17ca54_characterizeImage_947523_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6cf5b653-a70d-4f3b-a80b-c492fb17ca54_characterizeImage_947523_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f15bfe16-cae1-4a6d-8677-79b2e2292eb0_isr_947523_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/f15bfe16-cae1-4a6d-8677-79b2e2292eb0_isr_947523_51.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f73b77a8d54989b73e5ac1a3c832be1 with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash ef484a6ddcb6c05b215e89f4ff9ceb49
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 732
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/396c8a57-0532-4a2b-885d-86db6f159e94_characterizeImage_947528_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/396c8a57-0532-4a2b-885d-86db6f159e94_characterizeImage_947528_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8f3d1aa2-0d2f-44ec-9623-6b0466efbb10_isr_947528_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/8f3d1aa2-0d2f-44ec-9623-6b0466efbb10_isr_947528_6.stderr
parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 246d1d68a5af9ca5a10040f5b9ff1d14 with result from task 189
parsl.dataflow.dflow DEBUG: Task 189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 190 has memoization hash dbe2e08ba5b2fd49b20a0e2bbef232ba
parsl.dataflow.memoization INFO: Task 190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 733
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1a11717-1059-4ca9-8f99-5661a92b2150_characterizeImage_947523_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1a11717-1059-4ca9-8f99-5661a92b2150_characterizeImage_947523_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7eb989fc-9b6a-46f5-892e-f0723bf68cb0_isr_947523_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7eb989fc-9b6a-46f5-892e-f0723bf68cb0_isr_947523_10.stderr
parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9caa394d86abfd5d05468810f7fcbea4 with result from task 528
parsl.dataflow.dflow DEBUG: Task 528 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 529 has memoization hash f4982153afe963884ddd2c989591847c
parsl.dataflow.memoization INFO: Task 529 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 734
parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d807620e-4b1b-44b2-b2c9-2d1454648b70_characterizeImage_947526_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d807620e-4b1b-44b2-b2c9-2d1454648b70_characterizeImage_947526_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d9e0820a-5adc-4640-ab5b-7b6b2ec519b3_isr_947526_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/d9e0820a-5adc-4640-ab5b-7b6b2ec519b3_isr_947526_19.stderr
parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3af8338daaed2884f5b90bcd7dcf78ec with result from task 828
parsl.dataflow.dflow DEBUG: Task 828 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 2801210b8dacdd2301420761917826a3
parsl.dataflow.memoization INFO: Task 829 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 735
parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/3bcf33d6-a23c-4c32-9eb2-bfd4f16bb310_characterizeImage_947420_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/3bcf33d6-a23c-4c32-9eb2-bfd4f16bb310_characterizeImage_947420_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/87b01ebc-c2c8-42ab-8fb2-b0ce87063ebe_isr_947420_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/87b01ebc-c2c8-42ab-8fb2-b0ce87063ebe_isr_947420_38.stderr
parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da9d5b5b1482d42bd1bcf0ca223162f6 with result from task 540
parsl.dataflow.dflow DEBUG: Task 540 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 7d1d231a55bced3b041d59decbdefeca
parsl.dataflow.memoization INFO: Task 541 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 736
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/209cd2db-ac72-4d51-9f4e-e415b59456e9_characterizeImage_947418_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/209cd2db-ac72-4d51-9f4e-e415b59456e9_characterizeImage_947418_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/4f606d98-a3cf-4eb1-b489-ab62d468b66f_isr_947418_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/4f606d98-a3cf-4eb1-b489-ab62d468b66f_isr_947418_9.stderr
parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry caff1112f0b7132c845aa2b521b61c50 with result from task 834
parsl.dataflow.dflow DEBUG: Task 834 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 835 has memoization hash da6bb99445a12341a0e1281811ed5d51
parsl.dataflow.memoization INFO: Task 835 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 737
parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7895281d-8d15-4e30-bd08-a965ac148931_characterizeImage_947418_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7895281d-8d15-4e30-bd08-a965ac148931_characterizeImage_947418_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e6e2e012-352a-4fe4-a3fe-98ebe3213504_isr_947418_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e6e2e012-352a-4fe4-a3fe-98ebe3213504_isr_947418_57.stderr
parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc32f49484edef966edc00dd004b3b3 with result from task 519
parsl.dataflow.dflow DEBUG: Task 519 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 6c36053e1771d3d43c41d62e0ddcf7ab
parsl.dataflow.memoization INFO: Task 520 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 738
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/707cfa30-aa44-456a-b682-33707c87cf4c_characterizeImage_947523_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/707cfa30-aa44-456a-b682-33707c87cf4c_characterizeImage_947523_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4063502f-a7eb-45c3-b8f8-04a58e0f5d85_isr_947523_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/4063502f-a7eb-45c3-b8f8-04a58e0f5d85_isr_947523_26.stderr
parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84e0d2378316c884da74a40f93df418d with result from task 813
parsl.dataflow.dflow DEBUG: Task 813 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 814 has memoization hash c2b9e610a639eeb4b37cc565282cd8bb
parsl.dataflow.memoization INFO: Task 814 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 739
parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6cdb79a0-0e21-4787-8e6c-e6d8168f1964_characterizeImage_947418_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6cdb79a0-0e21-4787-8e6c-e6d8168f1964_characterizeImage_947418_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e280fcc3-b7f2-42f0-ad4d-afe98fa93c3a_isr_947418_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e280fcc3-b7f2-42f0-ad4d-afe98fa93c3a_isr_947418_62.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8ea061340bfe0fce76de60599073fb with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash e30f35b52a85d9503dab935f373ec4ca
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 740
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1bb11128-8dad-4c29-a3af-0ff597cd523c_characterizeImage_947418_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1bb11128-8dad-4c29-a3af-0ff597cd523c_characterizeImage_947418_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5b425dc5-3050-4b25-a814-e0c6be4ea908_isr_947418_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5b425dc5-3050-4b25-a814-e0c6be4ea908_isr_947418_44.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c396c72b459c9f3d48659af9235cbb6c with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash ccc1b90db63911f6a3d57be861aeaae7
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 741
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4f448ad2-ff64-4f0d-a9fe-8d35c3717e60_characterizeImage_947528_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4f448ad2-ff64-4f0d-a9fe-8d35c3717e60_characterizeImage_947528_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2789f2ac-7510-4998-8bf0-323bab83b63c_isr_947528_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/2789f2ac-7510-4998-8bf0-323bab83b63c_isr_947528_47.stderr
parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ec0b31351fd84e510b149ff486c7376 with result from task 516
parsl.dataflow.dflow DEBUG: Task 516 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 517 has memoization hash d4bb5c44042d667061108fed2e4949bd
parsl.dataflow.memoization INFO: Task 517 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 742
parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/231e88ef-4965-49ca-a398-d22918e4340d_characterizeImage_947523_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/231e88ef-4965-49ca-a398-d22918e4340d_characterizeImage_947523_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c7c7a9b-8116-46d4-b4f4-7163592db4ce_isr_947523_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9c7c7a9b-8116-46d4-b4f4-7163592db4ce_isr_947523_24.stderr
parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e4b1f1db87b84d4beca9862b833eb45 with result from task 558
parsl.dataflow.dflow DEBUG: Task 558 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 3de70864477ae643f53373f248ce047a
parsl.dataflow.memoization INFO: Task 559 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 743
parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/91b0cb1d-2da7-4a38-babc-cbcfa1ba9a0d_characterizeImage_947528_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/91b0cb1d-2da7-4a38-babc-cbcfa1ba9a0d_characterizeImage_947528_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dc43e858-cb06-4fc1-8cf1-ef471d59ee5d_isr_947528_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/dc43e858-cb06-4fc1-8cf1-ef471d59ee5d_isr_947528_22.stderr
parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 388b6971821e65d36f18b832c464b3d7 with result from task 525
parsl.dataflow.dflow DEBUG: Task 525 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 20465c79565d1cb1d621e830bd98c1c6
parsl.dataflow.memoization INFO: Task 526 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 744
parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eb93afa5-c41f-4fec-b796-346f5960a388_characterizeImage_947418_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eb93afa5-c41f-4fec-b796-346f5960a388_characterizeImage_947418_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/49e0ba86-4807-41c3-938f-ddf4ee541d94_isr_947418_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/49e0ba86-4807-41c3-938f-ddf4ee541d94_isr_947418_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:58 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97cd14b5a3273be07cd9586a1ae0d48f with result from task 537
parsl.dataflow.dflow DEBUG: Task 537 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 1dd6abe3d1f3dac5c2a6e1c43af920ae
parsl.dataflow.memoization INFO: Task 538 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 745
parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/49d1dbc7-dc24-4fcd-a0f8-c7f1ef04c28a_characterizeImage_947526_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/49d1dbc7-dc24-4fcd-a0f8-c7f1ef04c28a_characterizeImage_947526_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c61815eb-df2a-4eac-b9c2-adadebe4966d_isr_947526_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/c61815eb-df2a-4eac-b9c2-adadebe4966d_isr_947526_50.stderr
parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 471b1027bfbd111bec6e8a5ca9886873 with result from task 195
parsl.dataflow.dflow DEBUG: Task 195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 196 has memoization hash d39751b343f99d9cc9df3b9bb8d52a3a
parsl.dataflow.memoization INFO: Task 196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 746
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e3260a34-25cb-49f2-9a2d-8e194df0fec4_characterizeImage_947420_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e3260a34-25cb-49f2-9a2d-8e194df0fec4_characterizeImage_947420_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5ddfd005-b13f-4d45-a33d-bd1f2526da5c_isr_947420_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5ddfd005-b13f-4d45-a33d-bd1f2526da5c_isr_947420_50.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce0fa4146fe4b53a0c4ff6bf352b4733 with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 0bfe0ddf18056d0ee0ffa0e92256462c
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 747
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d0852d6a-42ce-4049-8c88-995fe0f7ee1c_characterizeImage_947418_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d0852d6a-42ce-4049-8c88-995fe0f7ee1c_characterizeImage_947418_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/67de567e-a45f-4f1a-9d4d-a65242858998_isr_947418_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/67de567e-a45f-4f1a-9d4d-a65242858998_isr_947418_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:58 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec4eddd1d73e4d6c931f76fdaa78f5fe with result from task 546
parsl.dataflow.dflow DEBUG: Task 546 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 154e9bb4206d21a6c442242869bf5959
parsl.dataflow.memoization INFO: Task 547 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 748
parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b490c1e5-afec-457d-bcc8-b88142df2778_characterizeImage_947526_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b490c1e5-afec-457d-bcc8-b88142df2778_characterizeImage_947526_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6db09c80-333c-4981-8c41-b03300f0bffb_isr_947526_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/6db09c80-333c-4981-8c41-b03300f0bffb_isr_947526_47.stderr
parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b47599671b02a8ef99614756df59d03 with result from task 543
parsl.dataflow.dflow DEBUG: Task 543 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 544 has memoization hash bac6812e9f8bd84e39d98b772902be02
parsl.dataflow.memoization INFO: Task 544 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 749
parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/3038955f-0527-4148-973c-8ddd4bf3e628_characterizeImage_947418_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/3038955f-0527-4148-973c-8ddd4bf3e628_characterizeImage_947418_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c3428dc4-f56c-48eb-97f3-3b3be4208484_isr_947418_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/c3428dc4-f56c-48eb-97f3-3b3be4208484_isr_947418_7.stderr
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5a9931a127928acaf204e48dcddd60ed with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 3f52af58b802d5bd204f0182edd94d76
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 750
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/23fe6b0c-b06a-4016-84fc-52a27373cb9b_characterizeImage_947524_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/23fe6b0c-b06a-4016-84fc-52a27373cb9b_characterizeImage_947524_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/b8d007f2-e75a-4089-8126-11f802a05e1d_isr_947524_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/b8d007f2-e75a-4089-8126-11f802a05e1d_isr_947524_54.stderr
parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb8e751ab08305bad41e0f53d1f345b8 with result from task 831
parsl.dataflow.dflow DEBUG: Task 831 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 832 has memoization hash aedeca590610532ee0b0f6acf264f9e0
parsl.dataflow.memoization INFO: Task 832 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 751
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/f6fcca11-11ff-485a-8d63-5e069bc52883_characterizeImage_947418_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/f6fcca11-11ff-485a-8d63-5e069bc52883_characterizeImage_947418_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/836bdf8d-1719-4816-b0c5-6548241a9377_isr_947418_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/836bdf8d-1719-4816-b0c5-6548241a9377_isr_947418_52.stderr
parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66602433f59cb70e603c3cf2cf8847e5 with result from task 561
parsl.dataflow.dflow DEBUG: Task 561 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 717b7ccd08f94743a1514ec654e00336
parsl.dataflow.memoization INFO: Task 562 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 752
parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fa68a04b-7c81-4832-b923-feb86d23e519_characterizeImage_947528_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fa68a04b-7c81-4832-b923-feb86d23e519_characterizeImage_947528_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c1374fb3-3494-4f4c-850b-719ebefd720e_isr_947528_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/c1374fb3-3494-4f4c-850b-719ebefd720e_isr_947528_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:59 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf9281172da8bbe8e7bdccbd3992d288 with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash da498683e1bdecc64068b9d6a0c28778
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 753
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/cf81a863-2369-4877-adc8-934424f92f23_characterizeImage_947523_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/cf81a863-2369-4877-adc8-934424f92f23_characterizeImage_947523_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3ce0d6c-3d56-46a5-857f-b06309ef4dce_isr_947523_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3ce0d6c-3d56-46a5-857f-b06309ef4dce_isr_947523_59.stderr
parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa79203abfcb093eb473105be6431fe6 with result from task 840
parsl.dataflow.dflow DEBUG: Task 840 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 841 has memoization hash ea8fd94e29986e7f1ac5b9a1e29aaa3c
parsl.dataflow.memoization INFO: Task 841 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 754
parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d81daa5a-4713-487a-856a-88bf9141b86c_characterizeImage_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d81daa5a-4713-487a-856a-88bf9141b86c_characterizeImage_947526_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bc44b7f8-9b2b-403a-bb2a-0ea3da87b2a1_isr_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/bc44b7f8-9b2b-403a-bb2a-0ea3da87b2a1_isr_947526_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:59 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b2dfdc898ea7714967e21564a273ebff with result from task 687
parsl.dataflow.dflow DEBUG: Task 687 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 95722257aa7a54edb770810d5483800d
parsl.dataflow.memoization INFO: Task 688 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 755
parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3ac5ecd6-1c7d-4ba1-b4aa-e0d97d91de9c_characterizeImage_947526_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3ac5ecd6-1c7d-4ba1-b4aa-e0d97d91de9c_characterizeImage_947526_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/763d8556-4a05-4045-835f-60ec0c718de1_isr_947526_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/763d8556-4a05-4045-835f-60ec0c718de1_isr_947526_31.stderr
parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e51ee7de9300b01b534dd8fb7a8867d2 with result from task 693
parsl.dataflow.dflow DEBUG: Task 693 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 694 has memoization hash ac2a324dfc6ad6720b8894dc7178fbe0
parsl.dataflow.memoization INFO: Task 694 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 756
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/46a2ca61-b0b7-4eb2-89b6-75fd8975697e_characterizeImage_947422_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/46a2ca61-b0b7-4eb2-89b6-75fd8975697e_characterizeImage_947422_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/075952bc-6a80-4c63-a5e9-acdbac292d31_isr_947422_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/075952bc-6a80-4c63-a5e9-acdbac292d31_isr_947422_4.stderr
parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 508266edc39f99f4ca163795b9623a4b with result from task 705
parsl.dataflow.dflow DEBUG: Task 705 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 5dd3c7612ff98dfba8ffabdc24cad121
parsl.dataflow.memoization INFO: Task 706 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 757
parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8e5d6d7a-a819-4881-84cb-b1bc4ea11edd_characterizeImage_947422_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8e5d6d7a-a819-4881-84cb-b1bc4ea11edd_characterizeImage_947422_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9424d2f-107c-40dc-9aae-e747d9032972_isr_947422_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/f9424d2f-107c-40dc-9aae-e747d9032972_isr_947422_32.stderr
parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d254f20eddedfadaed01d1017c5347e with result from task 690
parsl.dataflow.dflow DEBUG: Task 690 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 254c5029e923516a951edea940eee6be
parsl.dataflow.memoization INFO: Task 691 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 758
parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb207f2b-3b32-4fdc-b642-36d142605abf_characterizeImage_947422_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb207f2b-3b32-4fdc-b642-36d142605abf_characterizeImage_947422_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ac75fc94-67ad-4f36-b9eb-f28b9533566a_isr_947422_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/ac75fc94-67ad-4f36-b9eb-f28b9533566a_isr_947422_62.stderr
parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5248263c1062636121a46dee75f6697 with result from task 708
parsl.dataflow.dflow DEBUG: Task 708 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 709 has memoization hash c022b5ed9b43da6794605ff3887fe102
parsl.dataflow.memoization INFO: Task 709 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 759
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ae01a9ff-e074-4369-9f52-48ecd84bfda2_characterizeImage_947526_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ae01a9ff-e074-4369-9f52-48ecd84bfda2_characterizeImage_947526_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/46eb6ebd-503b-435d-a8f3-7ba900c2f5b3_isr_947526_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/46eb6ebd-503b-435d-a8f3-7ba900c2f5b3_isr_947526_5.stderr
parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6ab75ab3c8a3ef00c8e459677ff4423 with result from task 699
parsl.dataflow.dflow DEBUG: Task 699 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 8e58dc1a4a8dbfb02c08eae97746c874
parsl.dataflow.memoization INFO: Task 700 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 760
parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/91409b30-97b6-4588-a1fe-88515c5b17ec_characterizeImage_947420_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/91409b30-97b6-4588-a1fe-88515c5b17ec_characterizeImage_947420_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47f6608e-1190-4440-8ebb-ce560a1e0e52_isr_947420_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/47f6608e-1190-4440-8ebb-ce560a1e0e52_isr_947420_1.stderr
parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 696393f570a0bf3aa609c2d6e46f0f64 with result from task 696
parsl.dataflow.dflow DEBUG: Task 696 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 98fa8ba238bf5b32bb085a930ebbf6ed
parsl.dataflow.memoization INFO: Task 697 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 761
parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08ec80a8-fa50-4649-a42d-ac7cab71aeae_characterizeImage_947526_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08ec80a8-fa50-4649-a42d-ac7cab71aeae_characterizeImage_947526_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1ab2b1d6-3583-499a-b0af-02646c0b4388_isr_947526_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/1ab2b1d6-3583-499a-b0af-02646c0b4388_isr_947526_20.stderr
parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b2bbd9881ca30efa125616988781d95 with result from task 702
parsl.dataflow.dflow DEBUG: Task 702 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 9ce99f5856d5a34471f5d10f016b4860
parsl.dataflow.memoization INFO: Task 703 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 762
parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/6a4f5c56-7199-46da-9e8e-336b287fc5b9_characterizeImage_947420_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/6a4f5c56-7199-46da-9e8e-336b287fc5b9_characterizeImage_947420_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f57f57b6-b782-4b84-97cd-ff27572bf432_isr_947420_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/f57f57b6-b782-4b84-97cd-ff27572bf432_isr_947420_33.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 238 try 0 failed
parsl.dataflow.dflow INFO: Task 238 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash fa4e6fda98cdbf97c82cbcee5d8f4308
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 1 launched on executor multi with executor id 763
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stderr
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59ebc80fea5cef5ac194e573a9eb2227 with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash aaaeb72ca18fff5e20d0543e57a6e1ee
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 764
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1c12f9cd-349c-4f5d-9384-f753542ff6f8_characterizeImage_947422_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1c12f9cd-349c-4f5d-9384-f753542ff6f8_characterizeImage_947422_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/07d0f5c8-769b-4696-b0a4-f3604df46e6b_isr_947422_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/07d0f5c8-769b-4696-b0a4-f3604df46e6b_isr_947422_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:26:59 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3fd432c48692e36827152c32b3174df1 with result from task 531
parsl.dataflow.dflow DEBUG: Task 531 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 532 has memoization hash ec43d5fe6bb83cf0fb63d482d047af20
parsl.dataflow.memoization INFO: Task 532 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 765
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/698887ce-cf85-43cb-8431-fa1e65dc8d80_characterizeImage_947528_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/698887ce-cf85-43cb-8431-fa1e65dc8d80_characterizeImage_947528_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b81c1d71-db3c-4cee-a84b-74eb18a80452_isr_947528_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b81c1d71-db3c-4cee-a84b-74eb18a80452_isr_947528_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:00 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fabcd511077fdd8e4562d641c3c199c4 with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash c3e7398a404d573bead796942502fbc7
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 766
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/917a10a4-bfcd-4892-95c1-5e357dcffdb7_characterizeImage_947420_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/917a10a4-bfcd-4892-95c1-5e357dcffdb7_characterizeImage_947420_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/abd83f4c-a7a9-4813-bd1f-3419a82cc305_isr_947420_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/abd83f4c-a7a9-4813-bd1f-3419a82cc305_isr_947420_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:00 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:01 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:01 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:01 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:02 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:02 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:03 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:03 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:03 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:04 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:04 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:04 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 898 try 0 failed
parsl.dataflow.dflow INFO: Task 898 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 9db612024c015dcb56eade0701f9db06
parsl.dataflow.memoization INFO: Task 898 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 898 try 1 launched on executor multi with executor id 767
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:05 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:05 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 373 try 0 failed
parsl.dataflow.dflow INFO: Task 373 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 373 has memoization hash f67452c2996569f4a88394b01260ceb8
parsl.dataflow.memoization INFO: Task 373 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 373 try 1 launched on executor multi with executor id 768
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 226 try 0 failed
parsl.dataflow.dflow INFO: Task 226 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash c0d2cd095fbb412acd645460259402fd
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 1 launched on executor multi with executor id 769
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 238 try 1 failed
parsl.dataflow.dflow ERROR: Task 238 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry fa4e6fda98cdbf97c82cbcee5d8f4308 with result from task 238
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 239 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 239 try 0 failed
parsl.dataflow.dflow INFO: Task 239 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/a289c74b-0dac-4b4c-8597-f2055e75d45f_calibrate_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/a289c74b-0dac-4b4c-8597-f2055e75d45f_calibrate_947420_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/05ee6a64-6390-48f9-b25c-40e5906f27bc_characterizeImage_947420_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:06 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 352 try 1 failed
parsl.dataflow.dflow ERROR: Task 352 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cfec2f9edbbcea86f056e813319bd42 with result from task 352
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 353 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 353 try 0 failed
parsl.dataflow.dflow INFO: Task 353 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/12332dd8-e75b-4026-b005-78168d9b1ffb_calibrate_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/12332dd8-e75b-4026-b005-78168d9b1ffb_calibrate_947422_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a84df57d-2583-4ffb-8e32-42fbe22a8707_characterizeImage_947422_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:06 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:07 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:07 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:07 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:08 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56659ec7fc545dcfc96e2f4e66ee3a5d with result from task 972
parsl.dataflow.dflow DEBUG: Task 972 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 0f0f09d42ac3963abdeb62ecc678af65
parsl.dataflow.memoization INFO: Task 973 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 770
parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6dd871e-45f8-4b11-bdf1-2381439d6caa_characterizeImage_947523_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6dd871e-45f8-4b11-bdf1-2381439d6caa_characterizeImage_947523_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6a2ae717-0668-4208-b3ce-ada8499b2764_isr_947523_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/6a2ae717-0668-4208-b3ce-ada8499b2764_isr_947523_28.stderr
parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3454512b34a0a62e80b256cf59555ff6 with result from task 969
parsl.dataflow.dflow DEBUG: Task 969 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 233a8ec01394f2967dd80e1d801977a5
parsl.dataflow.memoization INFO: Task 970 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 771
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d1ad0f62-8fea-4e81-aa7a-ed5805afa87d_characterizeImage_947422_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d1ad0f62-8fea-4e81-aa7a-ed5805afa87d_characterizeImage_947422_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/8fae28cb-2c44-432d-825b-01534f663e18_isr_947422_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/8fae28cb-2c44-432d-825b-01534f663e18_isr_947422_34.stderr
parsl.dataflow.dflow INFO: Task 1281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59ecb3d5bc483328b22ba132d5b6b538 with result from task 1281
parsl.dataflow.dflow DEBUG: Task 1281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1282 has memoization hash d3a86a0213e9d40965977f48285cc1a0
parsl.dataflow.memoization INFO: Task 1282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1282 try 0 launched on executor multi with executor id 772
parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/927ef615-cb14-4848-a1c8-f01c63223f31_characterizeImage_947523_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/927ef615-cb14-4848-a1c8-f01c63223f31_characterizeImage_947523_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/327cc6a0-e056-4973-bdc5-ae38ae661419_isr_947523_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/327cc6a0-e056-4973-bdc5-ae38ae661419_isr_947523_25.stderr
parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db3d3a51d05189e60588beca8e9ab550 with result from task 1029
parsl.dataflow.dflow DEBUG: Task 1029 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 1c3eba08dbbd386297da396b431648d7
parsl.dataflow.memoization INFO: Task 1030 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 773
parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a02621e1-b4a6-4216-95c6-fe311a04c979_characterizeImage_947528_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a02621e1-b4a6-4216-95c6-fe311a04c979_characterizeImage_947528_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/027cd8bb-5bc9-4011-b470-679a6c175884_isr_947528_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/027cd8bb-5bc9-4011-b470-679a6c175884_isr_947528_13.stderr
parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6523f3205a3e35edebc2ecfa37e86dda with result from task 1020
parsl.dataflow.dflow DEBUG: Task 1020 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash b7d81e722af09c6dfd397040ac216cc7
parsl.dataflow.memoization INFO: Task 1021 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 774
parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6a6cf7f1-4b2d-4567-8941-27ab00c568a0_characterizeImage_947418_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6a6cf7f1-4b2d-4567-8941-27ab00c568a0_characterizeImage_947418_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/02cbf624-9284-45f6-b962-a3d80f4b892f_isr_947418_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/02cbf624-9284-45f6-b962-a3d80f4b892f_isr_947418_46.stderr
parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f63b00e0c0bb4c14b986623afdb180c2 with result from task 1032
parsl.dataflow.dflow DEBUG: Task 1032 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 67abe870c46ea006d232c963128f1290
parsl.dataflow.memoization INFO: Task 1033 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 775
parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/c23b2dfd-514c-4888-a1b4-bd5a894ca416_characterizeImage_947422_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/c23b2dfd-514c-4888-a1b4-bd5a894ca416_characterizeImage_947422_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2c877851-b7fc-4f0f-9128-c84d057240a8_isr_947422_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/2c877851-b7fc-4f0f-9128-c84d057240a8_isr_947422_50.stderr
parsl.dataflow.dflow INFO: Task 1149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67bbcf49c6e91d921438f3063667ca4e with result from task 1149
parsl.dataflow.dflow DEBUG: Task 1149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1150 has memoization hash 675ee48fe7b8f2f76277a94d904c3d98
parsl.dataflow.memoization INFO: Task 1150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1150 try 0 launched on executor multi with executor id 776
parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8e5aaa63-e492-406f-9052-5834b75313ce_characterizeImage_947526_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8e5aaa63-e492-406f-9052-5834b75313ce_characterizeImage_947526_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/57aaf85a-ce82-4106-976c-0df6bb035413_isr_947526_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/57aaf85a-ce82-4106-976c-0df6bb035413_isr_947526_48.stderr
parsl.dataflow.dflow INFO: Task 1158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eee489a78d619cbf4a5203590f11f474 with result from task 1158
parsl.dataflow.dflow DEBUG: Task 1158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1159 has memoization hash 16a89b54f2590a690247d476f591ba3c
parsl.dataflow.memoization INFO: Task 1159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1159 try 0 launched on executor multi with executor id 777
parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6e7d9729-fe29-4ae4-bb51-ab0ab1e1633d_characterizeImage_947528_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6e7d9729-fe29-4ae4-bb51-ab0ab1e1633d_characterizeImage_947528_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1e8ca4da-8f92-49fe-b267-157da6a1ac90_isr_947528_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1e8ca4da-8f92-49fe-b267-157da6a1ac90_isr_947528_29.stderr
parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9970b4da2a25ddd174f4d033b6cbe0a9 with result from task 1011
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:08 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 1011 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 3e806ed525afbc62bb5ce9a1de3330e6
parsl.dataflow.memoization INFO: Task 1012 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 778
parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/84655c21-6494-43a7-9647-95db66ee33ac_characterizeImage_947528_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/84655c21-6494-43a7-9647-95db66ee33ac_characterizeImage_947528_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11237c37-a4a9-45b7-920f-a7ee8e806b0e_isr_947528_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/11237c37-a4a9-45b7-920f-a7ee8e806b0e_isr_947528_38.stderr
parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ad920edea67f29587b4bc7f6161f941 with result from task 1008
parsl.dataflow.dflow DEBUG: Task 1008 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash f1312f17fe20f7311240e2b3aa00fd3f
parsl.dataflow.memoization INFO: Task 1009 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 779
parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b0b7a480-6189-4c78-8e72-dc8a1b45b012_characterizeImage_947420_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b0b7a480-6189-4c78-8e72-dc8a1b45b012_characterizeImage_947420_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/36aceef4-2f39-43b2-b1cd-9f8d5efd5b28_isr_947420_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/36aceef4-2f39-43b2-b1cd-9f8d5efd5b28_isr_947420_6.stderr
parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed710c134bd33578e8c9cc0228ba8ef9 with result from task 993
parsl.dataflow.dflow DEBUG: Task 993 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 789471b23f8550e2d989d95aedf99cc7
parsl.dataflow.memoization INFO: Task 994 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 780
parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/dae1280d-d1e0-4ccf-9125-9aedc8b6a574_characterizeImage_947420_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/dae1280d-d1e0-4ccf-9125-9aedc8b6a574_characterizeImage_947420_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/58b45435-e135-4d0a-a5e9-45ec10a1effa_isr_947420_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/58b45435-e135-4d0a-a5e9-45ec10a1effa_isr_947420_17.stderr
parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04dcf467f2b3bfe04c55838b1f9d9eae with result from task 999
parsl.dataflow.dflow DEBUG: Task 999 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 711ce2444e91d9ad8b1ef698e82559c0
parsl.dataflow.memoization INFO: Task 1000 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 781
parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ce772f1f-5c69-4401-be44-50b6d15ec488_characterizeImage_947422_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ce772f1f-5c69-4401-be44-50b6d15ec488_characterizeImage_947422_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/20b32f99-3ce9-489b-971a-8e43e05c33bf_isr_947422_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/20b32f99-3ce9-489b-971a-8e43e05c33bf_isr_947422_15.stderr
parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf61b9b2f69fed5f3605980995c9cd6a with result from task 996
parsl.dataflow.dflow DEBUG: Task 996 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 564f194280f0b5eb132116275cd82ae3
parsl.dataflow.memoization INFO: Task 997 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 782
parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8cc4155c-21cb-489f-a2d4-9258ce35e250_characterizeImage_947418_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8cc4155c-21cb-489f-a2d4-9258ce35e250_characterizeImage_947418_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e5c432ad-6f35-4f0e-b546-2b382d213fe3_isr_947418_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/e5c432ad-6f35-4f0e-b546-2b382d213fe3_isr_947418_22.stderr
parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a5fad219b1b570e944d73ade232a210 with result from task 1038
parsl.dataflow.dflow DEBUG: Task 1038 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 6552b0f9955d7b9004d4da17545c9a7d
parsl.dataflow.memoization INFO: Task 1039 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 783
parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/976861c6-3757-438c-89d0-92dc16adcbd6_characterizeImage_947526_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/976861c6-3757-438c-89d0-92dc16adcbd6_characterizeImage_947526_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0746290a-d097-4b80-b294-e6ad12187d37_isr_947526_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0746290a-d097-4b80-b294-e6ad12187d37_isr_947526_41.stderr
parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d29198dc166182690e6e5a5fe34dd31a with result from task 1014
parsl.dataflow.dflow DEBUG: Task 1014 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash e8d559d6a30e3432af226799a8864c35
parsl.dataflow.memoization INFO: Task 1015 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 784
parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/31ba8635-6c06-444d-aea4-be0e3faa3c70_characterizeImage_947420_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/31ba8635-6c06-444d-aea4-be0e3faa3c70_characterizeImage_947420_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2ae1a682-b0dd-450c-86a0-7cd7ed3b7067_isr_947420_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2ae1a682-b0dd-450c-86a0-7cd7ed3b7067_isr_947420_40.stderr
parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2ca3da327baafa18553cb7db54c6e83 with result from task 1017
parsl.dataflow.dflow DEBUG: Task 1017 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 26469e31da9ce9e10d05d234eb263fc4
parsl.dataflow.memoization INFO: Task 1018 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 785
parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4e9185c4-5fba-4ee0-9bfe-fcf6660e1c94_characterizeImage_947420_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4e9185c4-5fba-4ee0-9bfe-fcf6660e1c94_characterizeImage_947420_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2c13f486-4e02-4279-a26a-e6ae0d51e50a_isr_947420_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2c13f486-4e02-4279-a26a-e6ae0d51e50a_isr_947420_5.stderr
parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b83c0ac100ab4d6b217f2083e1492f88 with result from task 1026
parsl.dataflow.dflow DEBUG: Task 1026 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash bfec3034b83a9c62f2eaa2bfb202b1ee
parsl.dataflow.memoization INFO: Task 1027 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 786
parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5e3343d9-53eb-40b3-89ff-310a75d49d24_characterizeImage_947418_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5e3343d9-53eb-40b3-89ff-310a75d49d24_characterizeImage_947418_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/39058b2f-4771-403c-93a1-825e28c63716_isr_947418_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/39058b2f-4771-403c-93a1-825e28c63716_isr_947418_11.stderr
parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ec06190d5e344ca1756cee7bcbaa433 with result from task 1005
parsl.dataflow.dflow DEBUG: Task 1005 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash 7677fd918ec71d9cb90d2b8d03ae24c7
parsl.dataflow.memoization INFO: Task 1006 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 787
parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2943865f-60cf-4932-8399-28543b421fd8_characterizeImage_947528_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2943865f-60cf-4932-8399-28543b421fd8_characterizeImage_947528_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/08ccaa4d-b0e2-4efe-ad6e-76e08d5040d0_isr_947528_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/08ccaa4d-b0e2-4efe-ad6e-76e08d5040d0_isr_947528_30.stderr
parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d089667f6f744ce83d6f5d74d5fc890 with result from task 1035
parsl.dataflow.dflow DEBUG: Task 1035 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 1970737f32fef91ebc57fa3e212ed595
parsl.dataflow.memoization INFO: Task 1036 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 788
parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7cfd55d0-fa93-4df0-9d5c-23d4c7714085_characterizeImage_947524_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7cfd55d0-fa93-4df0-9d5c-23d4c7714085_characterizeImage_947524_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/55220a37-7c5d-4d58-b1b6-2aaa3c43ee5d_isr_947524_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/55220a37-7c5d-4d58-b1b6-2aaa3c43ee5d_isr_947524_10.stderr
parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a3cfc32a46a9f33ae54ff70a14cbfd6c with result from task 1023
parsl.dataflow.dflow DEBUG: Task 1023 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 6dd84d28f5461375f018ea0c3827df06
parsl.dataflow.memoization INFO: Task 1024 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 789
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e2eca7fd-864a-46d4-9092-4b1ddf72e76b_characterizeImage_947418_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e2eca7fd-864a-46d4-9092-4b1ddf72e76b_characterizeImage_947418_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cc9061b-b6ed-45ab-b5f9-6b974ab94b9a_isr_947418_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/0cc9061b-b6ed-45ab-b5f9-6b974ab94b9a_isr_947418_20.stderr
parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 821af2f544fbdf7e4520e2103ef5b8de with result from task 921
parsl.dataflow.dflow DEBUG: Task 921 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 922 has memoization hash d77f6e687fc01db8a8ca7a6e9ebc8353
parsl.dataflow.memoization INFO: Task 922 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 790
parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/38492558-d307-45ac-a7c0-95537f9b8fc9_characterizeImage_947526_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/38492558-d307-45ac-a7c0-95537f9b8fc9_characterizeImage_947526_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8670acad-4de0-4d03-b091-7d5048df8e5f_isr_947526_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/8670acad-4de0-4d03-b091-7d5048df8e5f_isr_947526_51.stderr
parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e98732876376d98f838a4b173a8d7183 with result from task 924
parsl.dataflow.dflow DEBUG: Task 924 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 925 has memoization hash ba43e62c35349cdba8188ab54bde64cf
parsl.dataflow.memoization INFO: Task 925 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 791
parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1cd4ac1a-d24e-4b16-a981-894176d99a4a_characterizeImage_947524_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1cd4ac1a-d24e-4b16-a981-894176d99a4a_characterizeImage_947524_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce572fb3-5c92-4624-8eee-d2a9e283c548_isr_947524_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/ce572fb3-5c92-4624-8eee-d2a9e283c548_isr_947524_6.stderr
parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96fff5f2982df0ee4f65be857aa115e3 with result from task 927
parsl.dataflow.dflow DEBUG: Task 927 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 5850851e0538140dc7be7efa85d8fa0e
parsl.dataflow.memoization INFO: Task 928 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 792
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/90fc6dfa-9260-4425-bf98-3ffa5d033733_characterizeImage_947523_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/90fc6dfa-9260-4425-bf98-3ffa5d033733_characterizeImage_947523_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/3895ab8b-10cd-4f97-8246-b3f63bb679da_isr_947523_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/3895ab8b-10cd-4f97-8246-b3f63bb679da_isr_947523_56.stderr
parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4571bc9a13dd825e1b26c096a5a5644b with result from task 933
parsl.dataflow.dflow DEBUG: Task 933 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 934 has memoization hash e0863d3819318a0d4d6a0ec39672f377
parsl.dataflow.memoization INFO: Task 934 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 793
parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bc3e1851-575a-4d68-8ad9-cb2136ce6804_characterizeImage_947526_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bc3e1851-575a-4d68-8ad9-cb2136ce6804_characterizeImage_947526_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b936ebc8-307c-44c6-a339-1ac91910b072_isr_947526_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b936ebc8-307c-44c6-a339-1ac91910b072_isr_947526_35.stderr
parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d92967c49995b76ce068ada78b355c4 with result from task 939
parsl.dataflow.dflow DEBUG: Task 939 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 940 has memoization hash c8da15ab05e6048ad5b42255aa06f65e
parsl.dataflow.memoization INFO: Task 940 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 794
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4d433184-603d-44b5-8e24-89c75163c292_characterizeImage_947524_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4d433184-603d-44b5-8e24-89c75163c292_characterizeImage_947524_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/77e23278-023c-4876-9852-76cad3b6132c_isr_947524_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/77e23278-023c-4876-9852-76cad3b6132c_isr_947524_19.stderr
parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d86a8d93a5f519cb1b6c876bd94cab1 with result from task 918
parsl.dataflow.dflow DEBUG: Task 918 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 64273d1ea8e62f46bff09b5230b5b3fb
parsl.dataflow.memoization INFO: Task 919 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 795
parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/06544cc6-1c0c-4a69-b463-68286ab9f1de_characterizeImage_947422_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/06544cc6-1c0c-4a69-b463-68286ab9f1de_characterizeImage_947422_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/65bd3b89-01eb-4d30-be64-7835de430cfd_isr_947422_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/65bd3b89-01eb-4d30-be64-7835de430cfd_isr_947422_20.stderr
parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8fc1a66998367b66b0f3164534d3543 with result from task 936
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:09 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 936 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 937 has memoization hash a2be91291622d8c681e678de44778e06
parsl.dataflow.memoization INFO: Task 937 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 796
parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a32d3dd-a04c-436c-8da4-d20c59623366_characterizeImage_947524_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a32d3dd-a04c-436c-8da4-d20c59623366_characterizeImage_947524_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dfc42436-56e1-4964-8b7a-ca3ce5da2b7e_isr_947524_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/dfc42436-56e1-4964-8b7a-ca3ce5da2b7e_isr_947524_26.stderr
parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16441f5f25af7dd7d77d08a2e2197293 with result from task 930
parsl.dataflow.dflow DEBUG: Task 930 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 931 has memoization hash fde4021bddcb625558adef32e06c45f4
parsl.dataflow.memoization INFO: Task 931 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 797
parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8dcee7ba-041f-47ed-acab-171a0a118444_characterizeImage_947523_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8dcee7ba-041f-47ed-acab-171a0a118444_characterizeImage_947523_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/77192c21-e693-4289-80f1-1defc2b1224f_isr_947523_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/77192c21-e693-4289-80f1-1defc2b1224f_isr_947523_39.stderr
parsl.dataflow.dflow INFO: Task 1134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3179a7e48a74379245b6674b1d5566c with result from task 1134
parsl.dataflow.dflow DEBUG: Task 1134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1135 has memoization hash 35d6a97e0e54d8d26f92f6a38192a83a
parsl.dataflow.memoization INFO: Task 1135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1135 try 0 launched on executor multi with executor id 798
parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b521c164-b0a7-4aa2-9243-db6fb3b23e92_characterizeImage_947422_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b521c164-b0a7-4aa2-9243-db6fb3b23e92_characterizeImage_947422_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/34ea5f45-5718-493e-b824-9158392eb203_isr_947422_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/34ea5f45-5718-493e-b824-9158392eb203_isr_947422_18.stderr
parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 192d44ffb5a578a9c43cdbea80d00d76 with result from task 942
parsl.dataflow.dflow DEBUG: Task 942 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 8b117767e06d2fe762122b51c765fc48
parsl.dataflow.memoization INFO: Task 943 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 799
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/c1f70494-9dc5-4f90-8691-048674f13407_characterizeImage_947528_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/c1f70494-9dc5-4f90-8691-048674f13407_characterizeImage_947528_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/85ca4416-1706-47fe-bc21-d8fc5762b6ef_isr_947528_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/85ca4416-1706-47fe-bc21-d8fc5762b6ef_isr_947528_51.stderr
parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88ac2184b92ba20eb73b0f9f6321bbab with result from task 945
parsl.dataflow.dflow DEBUG: Task 945 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 98bc227224f0777b29cb1315fdb058bf
parsl.dataflow.memoization INFO: Task 946 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 800
parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/21b1b8ec-3568-4946-8a4b-7b229cd385eb_characterizeImage_947418_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/21b1b8ec-3568-4946-8a4b-7b229cd385eb_characterizeImage_947418_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1fcf5a69-01ec-44e3-aea7-a9c647a74fe9_isr_947418_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/1fcf5a69-01ec-44e3-aea7-a9c647a74fe9_isr_947418_8.stderr
parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64f8662cc2836192739d0e71e0ad6644 with result from task 954
parsl.dataflow.dflow DEBUG: Task 954 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 955 has memoization hash e8bfd1bcd166990459cfc68944339b38
parsl.dataflow.memoization INFO: Task 955 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 801
parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/76a474de-ff67-462e-9cd8-75e0fdd19290_characterizeImage_947420_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/76a474de-ff67-462e-9cd8-75e0fdd19290_characterizeImage_947420_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/610da794-7c47-4d6b-b374-191cdfb2abc3_isr_947420_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/610da794-7c47-4d6b-b374-191cdfb2abc3_isr_947420_10.stderr
parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16a10f63f608000a832c5b5b653803b8 with result from task 948
parsl.dataflow.dflow DEBUG: Task 948 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 345feffb4c6f85ad349893bc29d93d8a
parsl.dataflow.memoization INFO: Task 949 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 802
parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/da32664d-e39c-4608-af92-02fdbedcf32c_characterizeImage_947420_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/da32664d-e39c-4608-af92-02fdbedcf32c_characterizeImage_947420_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/379c4a0a-9d1a-43ca-874f-c92c9171f8b5_isr_947420_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/379c4a0a-9d1a-43ca-874f-c92c9171f8b5_isr_947420_47.stderr
parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 05f376eef1dae36857a3b33b91c9687d with result from task 963
parsl.dataflow.dflow DEBUG: Task 963 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 632c22e3fb69c55fb0b6fab64febf280
parsl.dataflow.memoization INFO: Task 964 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 803
parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/df1ffd64-003d-4d12-9e21-ebd7bba867de_characterizeImage_947523_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/df1ffd64-003d-4d12-9e21-ebd7bba867de_characterizeImage_947523_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7e500f38-db63-4e56-b3b1-eba23cfa84a3_isr_947523_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7e500f38-db63-4e56-b3b1-eba23cfa84a3_isr_947523_35.stderr
parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6705e8f76f886e6bf2edff7e8afc36b with result from task 957
parsl.dataflow.dflow DEBUG: Task 957 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 958 has memoization hash cf51c3583b48bcd7579290cf1c85719c
parsl.dataflow.memoization INFO: Task 958 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 804
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a7b24683-7b68-4901-9e4f-9f718e6dd247_characterizeImage_947524_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a7b24683-7b68-4901-9e4f-9f718e6dd247_characterizeImage_947524_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/fc7e0f63-7f20-4dc1-be72-9435a1b8e8f1_isr_947524_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/fc7e0f63-7f20-4dc1-be72-9435a1b8e8f1_isr_947524_38.stderr
parsl.dataflow.dflow INFO: Task 1122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0b09656de8fab465b2e30f40248979f with result from task 1122
parsl.dataflow.dflow DEBUG: Task 1122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1123 has memoization hash db84556e20fcd3d40b6ee53d048521ae
parsl.dataflow.memoization INFO: Task 1123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1123 try 0 launched on executor multi with executor id 805
parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/35eeb198-2d1a-4a9b-b570-082ed8ed2179_characterizeImage_947418_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/35eeb198-2d1a-4a9b-b570-082ed8ed2179_characterizeImage_947418_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/492987d5-0a90-468a-b551-ccadbf082746_isr_947418_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/492987d5-0a90-468a-b551-ccadbf082746_isr_947418_23.stderr
parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7968439decac3b82088885277e548216 with result from task 951
parsl.dataflow.dflow DEBUG: Task 951 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 9293184c3e88f28bacada7d62d177837
parsl.dataflow.memoization INFO: Task 952 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 806
parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5d4b3362-5ac0-4119-97ce-6bc36d4fddd9_characterizeImage_947523_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5d4b3362-5ac0-4119-97ce-6bc36d4fddd9_characterizeImage_947523_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/da8d0138-1ad2-46cf-9c34-f49e6a842072_isr_947523_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/da8d0138-1ad2-46cf-9c34-f49e6a842072_isr_947523_60.stderr
parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4d1bdc62047074676cd2254663330dc with result from task 960
parsl.dataflow.dflow DEBUG: Task 960 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 93ca71e254b199f326562a856795cd0e
parsl.dataflow.memoization INFO: Task 961 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 807
parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/59bfa5b2-ac34-4741-8a19-ffd8c60b869f_characterizeImage_947528_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/59bfa5b2-ac34-4741-8a19-ffd8c60b869f_characterizeImage_947528_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6f3bbcf9-4680-4265-80a3-3c5638a82425_isr_947528_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/6f3bbcf9-4680-4265-80a3-3c5638a82425_isr_947528_42.stderr
parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95496af3bc06f3cdcd7dc49c683786b5 with result from task 990
parsl.dataflow.dflow DEBUG: Task 990 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 7355a57899a4a5db1511d44b9586bd32
parsl.dataflow.memoization INFO: Task 991 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 808
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/5d5260e0-981f-4ddd-933a-f54628745362_characterizeImage_947524_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/5d5260e0-981f-4ddd-933a-f54628745362_characterizeImage_947524_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/8818464d-1c96-4760-9792-3be3dd17fd29_isr_947524_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/8818464d-1c96-4760-9792-3be3dd17fd29_isr_947524_25.stderr
parsl.dataflow.dflow INFO: Task 1131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 046fe30d94b2a03f72d83d0a4ff9945c with result from task 1131
parsl.dataflow.dflow DEBUG: Task 1131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1132 has memoization hash 8f8c34c43fbe848abbaeb04d0081e600
parsl.dataflow.memoization INFO: Task 1132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1132 try 0 launched on executor multi with executor id 809
parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9ae03857-727f-4791-9d71-5d75367d8f05_characterizeImage_947418_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9ae03857-727f-4791-9d71-5d75367d8f05_characterizeImage_947418_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a0c9aa73-8373-4555-8d38-a8838abdb12a_isr_947418_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/a0c9aa73-8373-4555-8d38-a8838abdb12a_isr_947418_6.stderr
parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b303f91ca2b908c664fc79981263f3f with result from task 1002
parsl.dataflow.dflow DEBUG: Task 1002 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 402121cfbcf84075e0804fe20ec6ac91
parsl.dataflow.memoization INFO: Task 1003 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 810
parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/3aa34a38-02d1-4507-8394-90c5fb5a4500_characterizeImage_947524_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/3aa34a38-02d1-4507-8394-90c5fb5a4500_characterizeImage_947524_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/1945028b-703d-49c6-a03d-924bc1fdaa72_isr_947524_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/1945028b-703d-49c6-a03d-924bc1fdaa72_isr_947524_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:09 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b98ba387f7fbb9ea9f49de77c19fe1d3 with result from task 1143
parsl.dataflow.dflow DEBUG: Task 1143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1144 has memoization hash 3793bde8e7b7cfdff03b60262ceba4be
parsl.dataflow.memoization INFO: Task 1144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1144 try 0 launched on executor multi with executor id 811
parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/88d63cb6-78cd-4229-b4b4-a312b235bef6_characterizeImage_947528_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/88d63cb6-78cd-4229-b4b4-a312b235bef6_characterizeImage_947528_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/05b1e838-c5f6-4485-90a6-39ac4a009807_isr_947528_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/05b1e838-c5f6-4485-90a6-39ac4a009807_isr_947528_17.stderr
parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2c1910446a1d230a823071235aa0e51 with result from task 975
parsl.dataflow.dflow DEBUG: Task 975 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 6bb4b7fbacbe3acf0ecb1d8ec5996a29
parsl.dataflow.memoization INFO: Task 976 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 812
parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d50b4a27-056b-48e9-9725-575aa49e2b40_characterizeImage_947418_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d50b4a27-056b-48e9-9725-575aa49e2b40_characterizeImage_947418_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2d637ece-fc75-41a8-b3d6-15ba08daaa7f_isr_947418_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/2d637ece-fc75-41a8-b3d6-15ba08daaa7f_isr_947418_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:09 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f3d82093176ae955fcc2e69f750e9621 with result from task 1218
parsl.dataflow.dflow DEBUG: Task 1218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1219 has memoization hash 8eaaf68106f2988a70fc862659120d10
parsl.dataflow.memoization INFO: Task 1219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1219 try 0 launched on executor multi with executor id 813
parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ff5aa620-879f-4f13-b469-3007508d0cad_characterizeImage_947526_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ff5aa620-879f-4f13-b469-3007508d0cad_characterizeImage_947526_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a1d11a12-05f5-48cd-b2c0-3c077f031733_isr_947526_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/a1d11a12-05f5-48cd-b2c0-3c077f031733_isr_947526_46.stderr
parsl.dataflow.dflow INFO: Task 1200 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33da2977be7f8920f78aecab98a8a845 with result from task 1200
parsl.dataflow.dflow DEBUG: Task 1200 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1201 has memoization hash 59248b5439a711f4cb578ff3bb7cec2e
parsl.dataflow.memoization INFO: Task 1201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1201 try 0 launched on executor multi with executor id 814
parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5021c100-2883-4c19-8811-f101929ea69c_characterizeImage_947528_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5021c100-2883-4c19-8811-f101929ea69c_characterizeImage_947528_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4a8f353f-37f8-40b6-ab38-aed4f2de1744_isr_947528_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4a8f353f-37f8-40b6-ab38-aed4f2de1744_isr_947528_57.stderr
parsl.dataflow.dflow INFO: Task 1161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e38c343c0d4f090bbcb94680144ed72e with result from task 1161
parsl.dataflow.dflow DEBUG: Task 1161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1162 has memoization hash 0c4ee8865756773dc045776039c1a217
parsl.dataflow.memoization INFO: Task 1162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1162 try 0 launched on executor multi with executor id 815
parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3902428e-069a-4ce4-8cda-20df43447e09_characterizeImage_947526_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3902428e-069a-4ce4-8cda-20df43447e09_characterizeImage_947526_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f9deedfb-cc13-402b-8d48-2aba05dcdaf6_isr_947526_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/f9deedfb-cc13-402b-8d48-2aba05dcdaf6_isr_947526_18.stderr
parsl.dataflow.dflow INFO: Task 1164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c30efefeba6ae676f7bd0b13b384ab58 with result from task 1164
parsl.dataflow.dflow DEBUG: Task 1164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1165 has memoization hash aeb1a13e37c212e07e4135cb537a36cb
parsl.dataflow.memoization INFO: Task 1165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1165 try 0 launched on executor multi with executor id 816
parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08bdd8da-a653-41d2-9d0a-73d5646b5a14_characterizeImage_947526_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08bdd8da-a653-41d2-9d0a-73d5646b5a14_characterizeImage_947526_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9273f92f-3917-47e8-853d-d14254b63d2c_isr_947526_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/9273f92f-3917-47e8-853d-d14254b63d2c_isr_947526_23.stderr
parsl.dataflow.dflow INFO: Task 1293 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c9b3c71ea3a0c36840a0f4b7e3c6116 with result from task 1293
parsl.dataflow.dflow DEBUG: Task 1293 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1294 has memoization hash c429b5f14bdbd6ed0f9a0a3d2c135b6f
parsl.dataflow.memoization INFO: Task 1294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1294 try 0 launched on executor multi with executor id 817
parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e0bf357-70db-453a-9e6c-d06cf3338a6f_characterizeImage_947528_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e0bf357-70db-453a-9e6c-d06cf3338a6f_characterizeImage_947528_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/db247818-917a-4e58-a4ef-b158bba56cec_isr_947528_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/db247818-917a-4e58-a4ef-b158bba56cec_isr_947528_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:10 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 553 try 0 failed
parsl.dataflow.dflow INFO: Task 553 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 8c2ededd99d6d8fd90be60326e2e9e35
parsl.dataflow.memoization INFO: Task 553 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 553 try 1 launched on executor multi with executor id 818
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stderr
parsl.dataflow.dflow INFO: Task 1287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0811b4a6ba1f9bed9e23230c9a342a34 with result from task 1287
parsl.dataflow.dflow DEBUG: Task 1287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1288 has memoization hash dd0a75d25722064a4bf335782d5a172c
parsl.dataflow.memoization INFO: Task 1288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1288 try 0 launched on executor multi with executor id 819
parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3453b653-d8a1-4bb9-a2b2-4eac1e106db0_characterizeImage_947523_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3453b653-d8a1-4bb9-a2b2-4eac1e106db0_characterizeImage_947523_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0e242350-6191-48b7-949f-c6e0f3f83a14_isr_947523_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/0e242350-6191-48b7-949f-c6e0f3f83a14_isr_947523_14.stderr
parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7f3b069419fb5e848a7a5563d6f4364 with result from task 981
parsl.dataflow.dflow DEBUG: Task 981 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 982 has memoization hash df5be1dda22ff2aa4c12e9ea8bbec001
parsl.dataflow.memoization INFO: Task 982 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 820
parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/540f099a-1a67-4db7-a973-5caee3fb823f_characterizeImage_947420_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/540f099a-1a67-4db7-a973-5caee3fb823f_characterizeImage_947420_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3a5fa96e-70f7-4318-a76a-b45d7894df4c_isr_947420_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/3a5fa96e-70f7-4318-a76a-b45d7894df4c_isr_947420_29.stderr
parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e9812c41e53c00d8a6f0613e5cedb91 with result from task 978
parsl.dataflow.dflow DEBUG: Task 978 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 979 has memoization hash ecf35993b70943de220bab557c9c44c2
parsl.dataflow.memoization INFO: Task 979 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 821
parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6be81840-417b-4eab-9dd6-3c3618534e9f_characterizeImage_947528_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6be81840-417b-4eab-9dd6-3c3618534e9f_characterizeImage_947528_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/14a31a42-97af-463a-95ae-243b022b261a_isr_947528_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/14a31a42-97af-463a-95ae-243b022b261a_isr_947528_50.stderr
parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f685c5c6749c5134a9dd7d82ac30d36 with result from task 987
parsl.dataflow.dflow DEBUG: Task 987 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 988 has memoization hash ffffa400dadfa9522d61bf98ab675de3
parsl.dataflow.memoization INFO: Task 988 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 822
parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/c6f13e59-55b0-4c18-9b13-c38a3629baed_characterizeImage_947420_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/c6f13e59-55b0-4c18-9b13-c38a3629baed_characterizeImage_947420_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c2126688-3c5f-40c6-8af8-1b09901651eb_isr_947420_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/c2126688-3c5f-40c6-8af8-1b09901651eb_isr_947420_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:10 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fba605b43113823a416d856cf4afccc1 with result from task 1125
parsl.dataflow.dflow DEBUG: Task 1125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1126 has memoization hash 1a372ae0d660fa27d421c25cd24c4458
parsl.dataflow.memoization INFO: Task 1126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1126 try 0 launched on executor multi with executor id 823
parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/93d2eaad-4d49-45e3-9343-7009033e4be7_characterizeImage_947422_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/93d2eaad-4d49-45e3-9343-7009033e4be7_characterizeImage_947422_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0edd9aad-6198-40f7-b475-aef4cd3e33f1_isr_947422_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/0edd9aad-6198-40f7-b475-aef4cd3e33f1_isr_947422_25.stderr
parsl.dataflow.dflow INFO: Task 1167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad8395a20769f10258329eb9d07f5e03 with result from task 1167
parsl.dataflow.dflow DEBUG: Task 1167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1168 has memoization hash 4f5ca096af1f690984cfab3cd0bd50e3
parsl.dataflow.memoization INFO: Task 1168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1168 try 0 launched on executor multi with executor id 824
parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d5e82a8-9a73-4c4e-9190-737f8a5e96f0_characterizeImage_947528_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d5e82a8-9a73-4c4e-9190-737f8a5e96f0_characterizeImage_947528_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/69f18057-7f5c-4163-9b26-9409c3233160_isr_947528_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/69f18057-7f5c-4163-9b26-9409c3233160_isr_947528_35.stderr
parsl.dataflow.dflow INFO: Task 1140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 664d827242976580cf4919434540b14f with result from task 1140
parsl.dataflow.dflow DEBUG: Task 1140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1141 has memoization hash d88510f0ba6d6ea5ff8fe71579dee4f7
parsl.dataflow.memoization INFO: Task 1141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1141 try 0 launched on executor multi with executor id 825
parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/499f322b-ae0e-4700-b07a-76bbeea1c985_characterizeImage_947528_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/499f322b-ae0e-4700-b07a-76bbeea1c985_characterizeImage_947528_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b71d7abb-1230-4b2d-8ff4-b09c86850285_isr_947528_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/b71d7abb-1230-4b2d-8ff4-b09c86850285_isr_947528_55.stderr
parsl.dataflow.dflow INFO: Task 1146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16655c8047183f0e4d5cff275612a2ec with result from task 1146
parsl.dataflow.dflow DEBUG: Task 1146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1147 has memoization hash 6e1ffb4849fdf53ad5508c4295f41dad
parsl.dataflow.memoization INFO: Task 1147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1147 try 0 launched on executor multi with executor id 826
parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ea2c01ab-4303-4896-9489-4d9980fbf245_characterizeImage_947524_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ea2c01ab-4303-4896-9489-4d9980fbf245_characterizeImage_947524_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0cc3d9ce-d212-4df0-ba37-c5c4e16b0cc2_isr_947524_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/0cc3d9ce-d212-4df0-ba37-c5c4e16b0cc2_isr_947524_9.stderr
parsl.dataflow.dflow INFO: Task 1152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6a549fe000824c764c6fe919b6ef92e with result from task 1152
parsl.dataflow.dflow DEBUG: Task 1152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1153 has memoization hash 50240ac343a399374b245e86ec4c22e1
parsl.dataflow.memoization INFO: Task 1153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1153 try 0 launched on executor multi with executor id 827
parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/22db4a9d-cf42-4ea3-a0c6-33d177a9563e_characterizeImage_947528_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/22db4a9d-cf42-4ea3-a0c6-33d177a9563e_characterizeImage_947528_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/e5d69ca6-5678-4015-bf79-0e2abbae3b9c_isr_947528_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/e5d69ca6-5678-4015-bf79-0e2abbae3b9c_isr_947528_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:11 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28ed83bc08aeb106eb2faf7b91f663bb with result from task 1290
parsl.dataflow.dflow DEBUG: Task 1290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1291 has memoization hash 6adb2df28ffaf2fddb94a067c15da980
parsl.dataflow.memoization INFO: Task 1291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1291 try 0 launched on executor multi with executor id 828
parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cbc8fa64-58f4-4a91-92cc-310ccc67ca6f_characterizeImage_947524_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cbc8fa64-58f4-4a91-92cc-310ccc67ca6f_characterizeImage_947524_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/321cd90e-0530-4e0a-9bd0-b7e79e6d6f77_isr_947524_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/321cd90e-0530-4e0a-9bd0-b7e79e6d6f77_isr_947524_11.stderr
parsl.dataflow.dflow INFO: Task 1137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry abf2d9913bed6fb34dcb3e33a5adeb82 with result from task 1137
parsl.dataflow.dflow DEBUG: Task 1137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1138 has memoization hash 99c44c3eec041290479cd367177b4900
parsl.dataflow.memoization INFO: Task 1138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1138 try 0 launched on executor multi with executor id 829
parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4961b3a7-1c5b-4ddc-8fc8-a92af7036a0f_characterizeImage_947418_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4961b3a7-1c5b-4ddc-8fc8-a92af7036a0f_characterizeImage_947418_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5015b7e4-b22e-4e79-8f5c-c2708d27482c_isr_947418_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/5015b7e4-b22e-4e79-8f5c-c2708d27482c_isr_947418_36.stderr
parsl.dataflow.dflow INFO: Task 1128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f72bbf0801b540aa7206b5a986250db with result from task 1128
parsl.dataflow.dflow DEBUG: Task 1128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1129 has memoization hash e7e9021282d5883d3288ddb659455f29
parsl.dataflow.memoization INFO: Task 1129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1129 try 0 launched on executor multi with executor id 830
parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/128e275c-f516-4cb1-aff3-4b6951aa1dff_characterizeImage_947526_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/128e275c-f516-4cb1-aff3-4b6951aa1dff_characterizeImage_947526_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/187dd995-7b5d-4c95-8f47-553ce9c9171c_isr_947526_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/187dd995-7b5d-4c95-8f47-553ce9c9171c_isr_947526_58.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 226 try 1 failed
parsl.dataflow.dflow ERROR: Task 226 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry c0d2cd095fbb412acd645460259402fd with result from task 226
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 227 try 0 failed
parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/7d82cbed-d1ac-4d48-b596-040671d03018_calibrate_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/7d82cbed-d1ac-4d48-b596-040671d03018_calibrate_947523_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/c288f62e-6f46-411c-94d9-8ef57f14da3e_characterizeImage_947523_2.stderr
parsl.dataflow.dflow INFO: Task 1272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 435fe5565b542d74cc65fd720eb78107 with result from task 1272
parsl.dataflow.dflow DEBUG: Task 1272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1273 has memoization hash 6d555d1d15b9d512f470e0c7ddd6862d
parsl.dataflow.memoization INFO: Task 1273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1273 try 0 launched on executor multi with executor id 831
parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/39bfe967-4a53-4a79-8ad9-49935df3b8a6_characterizeImage_947528_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/39bfe967-4a53-4a79-8ad9-49935df3b8a6_characterizeImage_947528_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4932df2d-563f-470f-9875-518264dd86f8_isr_947528_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/4932df2d-563f-470f-9875-518264dd86f8_isr_947528_5.stderr
parsl.dataflow.dflow INFO: Task 1155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b103d04e3678dab0ce87d1431d240936 with result from task 1155
parsl.dataflow.dflow DEBUG: Task 1155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1156 has memoization hash a15631a6721063be1e9b4414c1573e22
parsl.dataflow.memoization INFO: Task 1156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1156 try 0 launched on executor multi with executor id 832
parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/943e68cf-f2d9-4195-bd65-3eefd3d15458_characterizeImage_947526_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/943e68cf-f2d9-4195-bd65-3eefd3d15458_characterizeImage_947526_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/cf778f2d-4d8d-49ec-8ff1-f4ae7b31ccf4_isr_947526_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/cf778f2d-4d8d-49ec-8ff1-f4ae7b31ccf4_isr_947526_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:11 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de20558d77329c58ba734d317d706dc0 with result from task 984
parsl.dataflow.dflow DEBUG: Task 984 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 985 has memoization hash c8c2eac24201a5e4656a8414fbac3265
parsl.dataflow.memoization INFO: Task 985 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 833
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/285dc637-1141-49ec-a71e-01a5ca6f9183_characterizeImage_947523_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/285dc637-1141-49ec-a71e-01a5ca6f9183_characterizeImage_947523_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d36df66-d5b3-4035-885c-650ab6531665_isr_947523_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d36df66-d5b3-4035-885c-650ab6531665_isr_947523_29.stderr
parsl.dataflow.dflow INFO: Task 1215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f1956afe0f32e43e55331036961e5563 with result from task 1215
parsl.dataflow.dflow DEBUG: Task 1215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1216 has memoization hash c623794dd617f419b410841660631efb
parsl.dataflow.memoization INFO: Task 1216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1216 try 0 launched on executor multi with executor id 834
parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/80a41f13-e67c-4514-b3c1-c009d06bfb2d_characterizeImage_947418_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/80a41f13-e67c-4514-b3c1-c009d06bfb2d_characterizeImage_947418_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b81ffc02-3e52-4b50-8faf-e260709ce2f9_isr_947418_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/b81ffc02-3e52-4b50-8faf-e260709ce2f9_isr_947418_26.stderr
parsl.dataflow.dflow INFO: Task 1209 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 113fe5f4d787d5fae27b3bf4e092098d with result from task 1209
parsl.dataflow.dflow DEBUG: Task 1209 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1210 has memoization hash 2057d7a625dbfd7d89c6cab00a14bde0
parsl.dataflow.memoization INFO: Task 1210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1210 try 0 launched on executor multi with executor id 835
parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f4d39cf1-ea41-4beb-b6be-14a6271560ce_characterizeImage_947526_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f4d39cf1-ea41-4beb-b6be-14a6271560ce_characterizeImage_947526_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/416e64e8-9f32-4d6a-a207-2ce3ffa01226_isr_947526_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/416e64e8-9f32-4d6a-a207-2ce3ffa01226_isr_947526_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:12 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:12 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4df1c179d02037fa3f576e560c16bb with result from task 1284
parsl.dataflow.dflow DEBUG: Task 1284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1285 has memoization hash 1a65ec887d6aba8f25e73f0209635b9a
parsl.dataflow.memoization INFO: Task 1285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1285 try 0 launched on executor multi with executor id 836
parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8b840617-2588-40e3-b751-c50f6caa858c_characterizeImage_947526_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8b840617-2588-40e3-b751-c50f6caa858c_characterizeImage_947526_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b8688102-fe1c-4428-bf7e-b59f1af19e5a_isr_947526_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/b8688102-fe1c-4428-bf7e-b59f1af19e5a_isr_947526_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:13 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2975998ba256bbb900ae5854cfea7f91 with result from task 1176
parsl.dataflow.dflow DEBUG: Task 1176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1177 has memoization hash adc7af201627e0b1f337301e17fd8336
parsl.dataflow.memoization INFO: Task 1177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1177 try 0 launched on executor multi with executor id 837
parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec62d6c9-ae9a-430c-b2b5-d3f9207d9af7_characterizeImage_947526_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec62d6c9-ae9a-430c-b2b5-d3f9207d9af7_characterizeImage_947526_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0647087a-77a8-426a-821c-ebc3f82c495e_isr_947526_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/0647087a-77a8-426a-821c-ebc3f82c495e_isr_947526_34.stderr
parsl.dataflow.dflow INFO: Task 1173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 66586965de63363794a8c706b82ea42e with result from task 1173
parsl.dataflow.dflow DEBUG: Task 1173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1174 has memoization hash 984efc46a630b671f16f9c05b78f9f5a
parsl.dataflow.memoization INFO: Task 1174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1174 try 0 launched on executor multi with executor id 838
parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ec32677c-d603-4688-9fc4-6389e72e311f_characterizeImage_947528_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ec32677c-d603-4688-9fc4-6389e72e311f_characterizeImage_947528_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/17d365dc-aa36-472b-b44b-fea993fea34e_isr_947528_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/17d365dc-aa36-472b-b44b-fea993fea34e_isr_947528_28.stderr
parsl.dataflow.dflow INFO: Task 1182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 129f7b4be228a1e3a3aee86c058dc3d8 with result from task 1182
parsl.dataflow.dflow DEBUG: Task 1182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1183 has memoization hash 718c17ef747d45e582d3e36056ba135b
parsl.dataflow.memoization INFO: Task 1183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1183 try 0 launched on executor multi with executor id 839
parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e87b88c-4c81-499c-a0e2-7deeaeba5faa_characterizeImage_947528_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e87b88c-4c81-499c-a0e2-7deeaeba5faa_characterizeImage_947528_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/27bff7e7-d582-479a-9696-25c923917ff1_isr_947528_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/27bff7e7-d582-479a-9696-25c923917ff1_isr_947528_15.stderr
parsl.dataflow.dflow INFO: Task 1185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bed0e4d0236aafa0ce751ffe4c86aca6 with result from task 1185
parsl.dataflow.dflow DEBUG: Task 1185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1186 has memoization hash 72271b45240caf79f84c4d4fe9e38806
parsl.dataflow.memoization INFO: Task 1186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1186 try 0 launched on executor multi with executor id 840
parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/488147ba-6dd6-4725-97dd-03ffae28e5d2_characterizeImage_947524_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/488147ba-6dd6-4725-97dd-03ffae28e5d2_characterizeImage_947524_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/22888194-e37e-4101-a08b-785cd98494d7_isr_947524_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/22888194-e37e-4101-a08b-785cd98494d7_isr_947524_35.stderr
parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3bec5f39013924f6a846299bf1cbfb1 with result from task 1107
parsl.dataflow.dflow DEBUG: Task 1107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 5a67aafe485aba39dcba33172c63e252
parsl.dataflow.memoization INFO: Task 1108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 841
parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a203b6f9-fbbe-45f5-ab53-24c1b5f3cf8c_characterizeImage_947418_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a203b6f9-fbbe-45f5-ab53-24c1b5f3cf8c_characterizeImage_947418_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d1cc225f-d2cc-427d-92b4-2af9c478e713_isr_947418_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/d1cc225f-d2cc-427d-92b4-2af9c478e713_isr_947418_40.stderr
parsl.dataflow.dflow INFO: Task 1116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6206e8002191f38a65109d15e4dcc3b2 with result from task 1116
parsl.dataflow.dflow DEBUG: Task 1116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1117 has memoization hash e8cc598c4c86f3402125b24cc589af82
parsl.dataflow.memoization INFO: Task 1117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1117 try 0 launched on executor multi with executor id 842
parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d9c8f0e7-9a07-464b-9a57-d6e27ac6eff3_characterizeImage_947422_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d9c8f0e7-9a07-464b-9a57-d6e27ac6eff3_characterizeImage_947422_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5036acce-23b5-4cbd-8ecb-cc1fb1239550_isr_947422_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/5036acce-23b5-4cbd-8ecb-cc1fb1239550_isr_947422_9.stderr
parsl.dataflow.dflow INFO: Task 1119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 283e5863a922796219400c4c0f8f913a with result from task 1119
parsl.dataflow.dflow DEBUG: Task 1119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1120 has memoization hash 1f1aa2e21d877d5fe72f7f98eacc2c0e
parsl.dataflow.memoization INFO: Task 1120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1120 try 0 launched on executor multi with executor id 843
parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/eaa03c38-3a1a-4f5e-ba79-3424d06ddece_characterizeImage_947526_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/eaa03c38-3a1a-4f5e-ba79-3424d06ddece_characterizeImage_947526_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5a92f97f-7806-4fd3-948b-39a10df55cdd_isr_947526_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/5a92f97f-7806-4fd3-948b-39a10df55cdd_isr_947526_45.stderr
parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d0e1b9aab5e7124dc61202f682c9123 with result from task 1104
parsl.dataflow.dflow DEBUG: Task 1104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 906eccbeb2076386a8e2681208ac5f6e
parsl.dataflow.memoization INFO: Task 1105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 844
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ba07cd9b-3fe7-4318-a38e-8f5125b423ac_characterizeImage_947422_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ba07cd9b-3fe7-4318-a38e-8f5125b423ac_characterizeImage_947422_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7984abcb-29dd-4f03-a0bd-db439191a2d8_isr_947422_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/7984abcb-29dd-4f03-a0bd-db439191a2d8_isr_947422_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:13 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:13 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b2f4c6cae672b6b4088275f8532c43a with result from task 1179
parsl.dataflow.dflow DEBUG: Task 1179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1180 has memoization hash b98d32da36496df39353d8fc01af3576
parsl.dataflow.memoization INFO: Task 1180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1180 try 0 launched on executor multi with executor id 845
parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a8e39e93-b5b5-4a1b-ab69-c2d454edafae_characterizeImage_947523_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a8e39e93-b5b5-4a1b-ab69-c2d454edafae_characterizeImage_947523_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fb97dcfe-0a08-4bea-848b-501039319d2c_isr_947523_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/fb97dcfe-0a08-4bea-848b-501039319d2c_isr_947523_12.stderr
parsl.dataflow.dflow INFO: Task 1194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e00c820086aba0dfb9a99ebe5b7a7fd with result from task 1194
parsl.dataflow.dflow DEBUG: Task 1194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1195 has memoization hash 68818b3105b61c739c37709c7047cbd8
parsl.dataflow.memoization INFO: Task 1195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1195 try 0 launched on executor multi with executor id 846
parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2d659a02-c77d-4596-a10f-06cd204411c9_characterizeImage_947523_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2d659a02-c77d-4596-a10f-06cd204411c9_characterizeImage_947523_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7927e069-94b0-4ffb-84c9-c4b8ea809d02_isr_947523_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/7927e069-94b0-4ffb-84c9-c4b8ea809d02_isr_947523_54.stderr
parsl.dataflow.dflow INFO: Task 1188 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e894053330332ca89867629e7e54ed3e with result from task 1188
parsl.dataflow.dflow DEBUG: Task 1188 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1189 has memoization hash 2c4973aa3d07fcee982c08a53e419e5a
parsl.dataflow.memoization INFO: Task 1189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1189 try 0 launched on executor multi with executor id 847
parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/867a1dfd-2e4c-438d-9b8c-669392b8088b_characterizeImage_947422_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/867a1dfd-2e4c-438d-9b8c-669392b8088b_characterizeImage_947422_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/3b9dd0a8-3078-4f02-a4b7-0e936e2e04cb_isr_947422_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/3b9dd0a8-3078-4f02-a4b7-0e936e2e04cb_isr_947422_7.stderr
parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7470ce5e6edf11c42cc7838cbf7ed10b with result from task 1113
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:14 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 1113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash 9db9c407317894f42f5dddca909c48d6
parsl.dataflow.memoization INFO: Task 1114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 848
parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec944901-3a17-4893-9c25-9ad129d24bd7_characterizeImage_947526_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec944901-3a17-4893-9c25-9ad129d24bd7_characterizeImage_947526_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/647396c0-fdf5-4fea-b643-8c679fb15bd8_isr_947526_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947526/647396c0-fdf5-4fea-b643-8c679fb15bd8_isr_947526_7.stderr
parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1457a09fd031adc1ba940024cf2b0d79 with result from task 1110
parsl.dataflow.dflow DEBUG: Task 1110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 12257bdaf3ca3c958bb478aa16e7cb49
parsl.dataflow.memoization INFO: Task 1111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 849
parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2089d1d5-aac9-407b-b666-9b5d0b527957_characterizeImage_947418_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2089d1d5-aac9-407b-b666-9b5d0b527957_characterizeImage_947418_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/6b904d7d-73e4-4e61-8bd6-724d3de36818_isr_947418_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/6b904d7d-73e4-4e61-8bd6-724d3de36818_isr_947418_50.stderr
parsl.dataflow.dflow INFO: Task 1278 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eaa7f5b90f5a9a146356406769ba731a with result from task 1278
parsl.dataflow.dflow DEBUG: Task 1278 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1279 has memoization hash 8268debf20b79a677af55da0f5062225
parsl.dataflow.memoization INFO: Task 1279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1279 try 0 launched on executor multi with executor id 850
parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af7e7348-0694-4f26-ada6-03bb4dcac92b_characterizeImage_947420_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af7e7348-0694-4f26-ada6-03bb4dcac92b_characterizeImage_947420_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5fb6c8b2-f0f1-4fa8-9b8c-48f6112a624b_isr_947420_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/5fb6c8b2-f0f1-4fa8-9b8c-48f6112a624b_isr_947420_44.stderr
parsl.dataflow.dflow INFO: Task 1275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0929be041cb86dbfc53751d3da97373d with result from task 1275
parsl.dataflow.dflow DEBUG: Task 1275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1276 has memoization hash 8026edbccd87cdf0e05f56135fde3468
parsl.dataflow.memoization INFO: Task 1276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1276 try 0 launched on executor multi with executor id 851
parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a813b9df-3966-4b1a-bb93-3b7f19f3ec80_characterizeImage_947422_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a813b9df-3966-4b1a-bb93-3b7f19f3ec80_characterizeImage_947422_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6865c994-e7fd-4771-ba08-bc47c99f43fb_isr_947422_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/6865c994-e7fd-4771-ba08-bc47c99f43fb_isr_947422_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:14 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9aa001273d41bd6ea3b3d4bbdd2622c with result from task 1101
parsl.dataflow.dflow DEBUG: Task 1101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 8df0f6b8531380101b7f62ed80de2468
parsl.dataflow.memoization INFO: Task 1102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 852
parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b407d6c6-2f4d-47c8-a650-4d16dbab7199_characterizeImage_947422_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b407d6c6-2f4d-47c8-a650-4d16dbab7199_characterizeImage_947422_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/be4c5dab-2e27-4562-9cbf-62c8c0422fc7_isr_947422_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/be4c5dab-2e27-4562-9cbf-62c8c0422fc7_isr_947422_3.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 898 try 1 failed
parsl.dataflow.dflow ERROR: Task 898 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 9db612024c015dcb56eade0701f9db06 with result from task 898
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 899 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 899 try 0 failed
parsl.dataflow.dflow INFO: Task 899 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0af582eb-059d-4016-a979-31e802d6cd48_calibrate_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0af582eb-059d-4016-a979-31e802d6cd48_calibrate_947526_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/45e4c934-2366-47c6-a323-2153d76bfccf_characterizeImage_947526_2.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 373 try 1 failed
parsl.dataflow.dflow ERROR: Task 373 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry f67452c2996569f4a88394b01260ceb8 with result from task 373
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 374 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 374 try 0 failed
parsl.dataflow.dflow INFO: Task 374 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:15 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1b5932ff-6a9f-4d9a-9a5c-e08c1e308503_calibrate_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1b5932ff-6a9f-4d9a-9a5c-e08c1e308503_calibrate_947528_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/aca60602-04cc-4b59-988a-c7e1b6df80de_characterizeImage_947528_2.stderr
parsl.dataflow.dflow INFO: Task 1191 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f0f6df96bec5b7a20a3f5e0286c23b9 with result from task 1191
parsl.dataflow.dflow DEBUG: Task 1191 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1192 has memoization hash 7c3583346baef57b8b83ea4e4b88958a
parsl.dataflow.memoization INFO: Task 1192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1192 try 0 launched on executor multi with executor id 853
parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2e293971-1040-42e4-b110-64babd4b25c7_isr_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/2e293971-1040-42e4-b110-64babd4b25c7_isr_947420_61.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 121 try 0 failed
parsl.dataflow.dflow INFO: Task 121 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 623cc275c57ff512fdc2fb588f16e794
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e967a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 1 launched on executor multi with executor id 854
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:15 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:16 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1203 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7ffb3d4d84d95e12169e3472288da00 with result from task 1203
parsl.dataflow.dflow DEBUG: Task 1203 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1204 has memoization hash f68527a2efbf9f233ba1883c75b69cac
parsl.dataflow.memoization INFO: Task 1204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1204 try 0 launched on executor multi with executor id 855
parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e7c4c10c-f0fa-47f6-90c0-fd1be9425810_characterizeImage_947420_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e7c4c10c-f0fa-47f6-90c0-fd1be9425810_characterizeImage_947420_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8ad6aee9-9ae9-400d-a466-59c258f09cbb_isr_947420_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947420/8ad6aee9-9ae9-400d-a466-59c258f09cbb_isr_947420_37.stderr
parsl.dataflow.dflow INFO: Task 1206 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60f66b5d36f6b9805eb34d1a6c81ab10 with result from task 1206
parsl.dataflow.dflow DEBUG: Task 1206 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1207 has memoization hash cc8cbbb05a50213d45ed1e4fa6f22498
parsl.dataflow.memoization INFO: Task 1207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1207 try 0 launched on executor multi with executor id 856
parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1d8e6da6-1ce3-4faf-8b49-2ccb126580ea_characterizeImage_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1d8e6da6-1ce3-4faf-8b49-2ccb126580ea_characterizeImage_947524_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/31e7cffa-c4ea-4905-982d-3d97d2b27538_isr_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/31e7cffa-c4ea-4905-982d-3d97d2b27538_isr_947524_61.stderr
parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d3aa1bcf1fbf61f5e6ebadb14ea1ad3 with result from task 1098
parsl.dataflow.dflow DEBUG: Task 1098 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash a459f6cad363c247ef98a2f466ffc0d1
parsl.dataflow.memoization INFO: Task 1099 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 857
parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/25a15a81-31dd-4d29-a258-6f087de5e17f_characterizeImage_947523_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/25a15a81-31dd-4d29-a258-6f087de5e17f_characterizeImage_947523_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d32aa41-421d-4d88-a06c-7f3e70e6ee8c_isr_947523_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/8d32aa41-421d-4d88-a06c-7f3e70e6ee8c_isr_947523_37.stderr
parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 917960c9fd1c9812da0e15fa93098f1d with result from task 913
parsl.dataflow.dflow DEBUG: Task 913 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 914 has memoization hash a538b3c16c650bf2398f5d3502ed5a64
parsl.dataflow.memoization INFO: Task 914 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 858
parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/70c1b9fe-67e1-46f4-881b-f126bc0bc6e0_calibrate_947418_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/70c1b9fe-67e1-46f4-881b-f126bc0bc6e0_calibrate_947418_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/537fe35e-2f10-4983-9f12-b17a199d9ad6_characterizeImage_947418_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/537fe35e-2f10-4983-9f12-b17a199d9ad6_characterizeImage_947418_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:16 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a127629c80cc1bedc15d429e8f163c66 with result from task 1212
parsl.dataflow.dflow DEBUG: Task 1212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1213 has memoization hash b871d15babc509616f94ac0b5a2d0f24
parsl.dataflow.memoization INFO: Task 1213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1213 try 0 launched on executor multi with executor id 859
parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/33a9a86c-182b-45e1-b84d-625303ec5c0d_characterizeImage_947524_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/33a9a86c-182b-45e1-b84d-625303ec5c0d_characterizeImage_947524_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9e666198-bcef-45fa-a7b1-fd8d6c9b0da0_isr_947524_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 1212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/9e666198-bcef-45fa-a7b1-fd8d6c9b0da0_isr_947524_59.stderr
parsl.dataflow.dflow INFO: Task 1197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b458c89bbec99d7735dafdfbc174e78 with result from task 1197
parsl.dataflow.dflow DEBUG: Task 1197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1198 has memoization hash 5fa975ac91ec5bdd4f76d628699a600a
parsl.dataflow.memoization INFO: Task 1198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:16 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Parsl task 1198 try 0 launched on executor multi with executor id 860
parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8e4cd71c-61a3-44d4-a32c-61beeaefb63e_isr_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/8e4cd71c-61a3-44d4-a32c-61beeaefb63e_isr_947418_61.stderr
parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6bbf48affe83937415e9a9c06006b669 with result from task 214
parsl.dataflow.dflow DEBUG: Task 214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 5c11dedd9148132f67961447281deb9f
parsl.dataflow.memoization INFO: Task 215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 861
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d6bea60c-aea1-4507-a42b-977e062c3b2f_calibrate_947418_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d6bea60c-aea1-4507-a42b-977e062c3b2f_calibrate_947418_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/31c1ff00-0114-4524-946a-f37fadad90d4_characterizeImage_947418_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/31c1ff00-0114-4524-946a-f37fadad90d4_characterizeImage_947418_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:17 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:17 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:18 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:18 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:19 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1239 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f16f6be98ebefcf5b45fd028ded7ed71 with result from task 1239
parsl.dataflow.dflow DEBUG: Task 1239 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1240 has memoization hash 695c8e43d10ca70d5232241e1833b22c
parsl.dataflow.memoization INFO: Task 1240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1240 try 0 launched on executor multi with executor id 862
parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/37814657-a34e-452c-8205-a6f48f920804_characterizeImage_947523_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/37814657-a34e-452c-8205-a6f48f920804_characterizeImage_947523_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9fc5494a-bce4-4f17-9800-3e01d9bfe5ae_isr_947523_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/9fc5494a-bce4-4f17-9800-3e01d9bfe5ae_isr_947523_3.stderr
parsl.dataflow.dflow INFO: Task 1221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e5bba796b7eb527905d2d261ad073eb with result from task 1221
parsl.dataflow.dflow DEBUG: Task 1221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1222 has memoization hash 768ac637e39d93cd2f1cef93215a4292
parsl.dataflow.memoization INFO: Task 1222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1222 try 0 launched on executor multi with executor id 863
parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/18521faf-71d9-459f-8348-9ec6092d8dc2_characterizeImage_947523_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/18521faf-71d9-459f-8348-9ec6092d8dc2_characterizeImage_947523_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/012df7b1-6c32-4062-b6c4-7d646ca5bab6_isr_947523_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/012df7b1-6c32-4062-b6c4-7d646ca5bab6_isr_947523_34.stderr
parsl.dataflow.dflow INFO: Task 1230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea8f042285f54dd82a31426630af401e with result from task 1230
parsl.dataflow.dflow DEBUG: Task 1230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1231 has memoization hash 6f0b714d92cdc6ba8cbe9881e8a5f946
parsl.dataflow.memoization INFO: Task 1231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1231 try 0 launched on executor multi with executor id 864
parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a1558d39-b089-4ee8-85f7-2728c83433c0_characterizeImage_947528_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a1558d39-b089-4ee8-85f7-2728c83433c0_characterizeImage_947528_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/0a0f642d-a9f1-4627-9909-bcb5f2ec6d38_isr_947528_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/0a0f642d-a9f1-4627-9909-bcb5f2ec6d38_isr_947528_46.stderr
parsl.dataflow.dflow INFO: Task 1248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 401b23d95415b756d585ae4982bef4db with result from task 1248
parsl.dataflow.dflow DEBUG: Task 1248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1249 has memoization hash 326029bb43d74d70444df6b670f5c4e1
parsl.dataflow.memoization INFO: Task 1249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1249 try 0 launched on executor multi with executor id 865
parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/33df8ce9-a0df-46ca-8fac-5ad26408e8a3_characterizeImage_947418_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/33df8ce9-a0df-46ca-8fac-5ad26408e8a3_characterizeImage_947418_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/415731bd-0501-419d-b8f9-152ab68f51bf_isr_947418_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/415731bd-0501-419d-b8f9-152ab68f51bf_isr_947418_37.stderr
parsl.dataflow.dflow INFO: Task 1257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c23c8f5c58a2867cf728d5276a1b9d5f with result from task 1257
parsl.dataflow.dflow DEBUG: Task 1257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1258 has memoization hash c1e17e2e974d3dee155c46eb5d17f328
parsl.dataflow.memoization INFO: Task 1258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1258 try 0 launched on executor multi with executor id 866
parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0a6c5ffe-029d-4c3f-8b47-869ec0a94d61_characterizeImage_947422_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0a6c5ffe-029d-4c3f-8b47-869ec0a94d61_characterizeImage_947422_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/56896d40-6cfd-4a95-8e7d-3e671c53ca3e_isr_947422_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/56896d40-6cfd-4a95-8e7d-3e671c53ca3e_isr_947422_42.stderr
parsl.dataflow.dflow INFO: Task 1245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87dd1035918938c05fa1323d16e6ab5a with result from task 1245
parsl.dataflow.dflow DEBUG: Task 1245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1246 has memoization hash 85de3f78e33b04569acbd9efd52a4de6
parsl.dataflow.memoization INFO: Task 1246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1246 try 0 launched on executor multi with executor id 867
parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/774b9d4f-3b32-483a-a801-d683e94f59da_characterizeImage_947418_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/774b9d4f-3b32-483a-a801-d683e94f59da_characterizeImage_947418_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/03086161-c6d2-49fd-a0eb-05696ff46ec3_isr_947418_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947418/03086161-c6d2-49fd-a0eb-05696ff46ec3_isr_947418_28.stderr
parsl.dataflow.dflow INFO: Task 1224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb0f8f3aad088ecaefdeaebef4982017 with result from task 1224
parsl.dataflow.dflow DEBUG: Task 1224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1225 has memoization hash ae735ec6d303db0f29038cfa71b38aed
parsl.dataflow.memoization INFO: Task 1225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1225 try 0 launched on executor multi with executor id 868
parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9375c84c-85df-42bf-aa8c-5de7b93d0b6e_characterizeImage_947524_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9375c84c-85df-42bf-aa8c-5de7b93d0b6e_characterizeImage_947524_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0cb6ccc-1fda-4af5-9482-95ed1cd8a5ea_isr_947524_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/f0cb6ccc-1fda-4af5-9482-95ed1cd8a5ea_isr_947524_21.stderr
parsl.dataflow.dflow INFO: Task 1227 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b2d9f750106cf18bc81c9fb41039a337 with result from task 1227
parsl.dataflow.dflow DEBUG: Task 1227 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1228 has memoization hash 0ca0d8f1372ffc4b52e1b62c82130550
parsl.dataflow.memoization INFO: Task 1228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1228 try 0 launched on executor multi with executor id 869
parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1d2f75f5-111f-45a3-809c-e898099845c3_characterizeImage_947528_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1d2f75f5-111f-45a3-809c-e898099845c3_characterizeImage_947528_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1478c42d-6177-4db8-bbcb-f998b66f50ac_isr_947528_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/1478c42d-6177-4db8-bbcb-f998b66f50ac_isr_947528_20.stderr
parsl.dataflow.dflow INFO: Task 1236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa7738ce040d309efdeaec5bf777287c with result from task 1236
parsl.dataflow.dflow DEBUG: Task 1236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1237 has memoization hash 5c56ce5f63e887c6757959b929ae9674
parsl.dataflow.memoization INFO: Task 1237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1237 try 0 launched on executor multi with executor id 870
parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7a9e012c-3549-4746-9848-0052e6c6614b_characterizeImage_947524_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7a9e012c-3549-4746-9848-0052e6c6614b_characterizeImage_947524_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5463bb8b-168c-4a77-82b0-398be0be8ab7_isr_947524_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/5463bb8b-168c-4a77-82b0-398be0be8ab7_isr_947524_32.stderr
parsl.dataflow.dflow INFO: Task 1233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b74ebbd09386ac7b2355882da62712a8 with result from task 1233
parsl.dataflow.dflow DEBUG: Task 1233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1234 has memoization hash 5d0c036e10511547b90f408da6e2a403
parsl.dataflow.memoization INFO: Task 1234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1234 try 0 launched on executor multi with executor id 871
parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6a983877-8c63-4c21-8c11-09e6a92eefd8_characterizeImage_947523_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6a983877-8c63-4c21-8c11-09e6a92eefd8_characterizeImage_947523_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3c1b9b5-1f32-446d-addd-24a1b8b21ae7_isr_947523_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947523/b3c1b9b5-1f32-446d-addd-24a1b8b21ae7_isr_947523_18.stderr
parsl.dataflow.dflow INFO: Task 1260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3cf6e8cc4ee6ed76e82a233c7e52cba4 with result from task 1260
parsl.dataflow.dflow DEBUG: Task 1260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1261 has memoization hash 44fe01d8882a3e7838f314f538ad5b48
parsl.dataflow.memoization INFO: Task 1261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1261 try 0 launched on executor multi with executor id 872
parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c93cbd81-ada3-4c5e-b47e-0488bd33f09b_characterizeImage_947524_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c93cbd81-ada3-4c5e-b47e-0488bd33f09b_characterizeImage_947524_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a4676c6-69ee-4a4d-ab7a-cbedc637ec8b_isr_947524_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/6a4676c6-69ee-4a4d-ab7a-cbedc637ec8b_isr_947524_52.stderr
parsl.dataflow.dflow INFO: Task 1251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe003e5faadcdfd51f01e4c4eddab595 with result from task 1251
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:19 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 1251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1252 has memoization hash 728f73f08291f259dc9d8684d31ee60f
parsl.dataflow.memoization INFO: Task 1252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1252 try 0 launched on executor multi with executor id 873
parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/120a6262-57ee-437a-9b97-e5e12bb1bc58_characterizeImage_947524_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/120a6262-57ee-437a-9b97-e5e12bb1bc58_characterizeImage_947524_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/508548d2-58d7-4de6-a502-135bb6bdcfaf_isr_947524_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/508548d2-58d7-4de6-a502-135bb6bdcfaf_isr_947524_50.stderr
parsl.dataflow.dflow INFO: Task 1266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8152dc0270cd29adc7ea9a9c8e86af8 with result from task 1266
parsl.dataflow.dflow DEBUG: Task 1266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1267 has memoization hash 15e4056fdc34c7db05493aad4fbdeb3f
parsl.dataflow.memoization INFO: Task 1267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1267 try 0 launched on executor multi with executor id 874
parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cf2411d9-6a0b-4775-bb89-5f45a153413f_characterizeImage_947528_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cf2411d9-6a0b-4775-bb89-5f45a153413f_characterizeImage_947528_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9877bae2-6764-42c7-b66a-f309977c88df_isr_947528_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/9877bae2-6764-42c7-b66a-f309977c88df_isr_947528_48.stderr
parsl.dataflow.dflow INFO: Task 1254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afdb2831a93973b5165a5702eaeb4c34 with result from task 1254
parsl.dataflow.dflow DEBUG: Task 1254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1255 has memoization hash 9311ca2e86f1461f0d30caa3f269d749
parsl.dataflow.memoization INFO: Task 1255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1255 try 0 launched on executor multi with executor id 875
parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6a0fe967-5da7-43be-827b-7d9e50e12f0c_characterizeImage_947524_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6a0fe967-5da7-43be-827b-7d9e50e12f0c_characterizeImage_947524_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e742706-1830-4a2e-9683-c8e63e61ad7c_isr_947524_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947524/2e742706-1830-4a2e-9683-c8e63e61ad7c_isr_947524_39.stderr
parsl.dataflow.dflow INFO: Task 1242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4e2e073f00062dfbf31bf6bf2dc06be with result from task 1242
parsl.dataflow.dflow DEBUG: Task 1242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1243 has memoization hash 3179ed1264cf69cc9adb7f37d9f7815e
parsl.dataflow.memoization INFO: Task 1243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1243 try 0 launched on executor multi with executor id 876
parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/51ff2bf5-648c-43e7-b4df-f66c1352a9df_characterizeImage_947528_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/51ff2bf5-648c-43e7-b4df-f66c1352a9df_characterizeImage_947528_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/831b7eb8-e38d-482c-bc13-44469ffd5d1f_isr_947528_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947528/831b7eb8-e38d-482c-bc13-44469ffd5d1f_isr_947528_43.stderr
parsl.dataflow.dflow INFO: Task 1263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3774f1cd012c028edfb57a926b6184ac with result from task 1263
parsl.dataflow.dflow DEBUG: Task 1263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1264 has memoization hash cc61103bcc9867cbc6a28f07899d705f
parsl.dataflow.memoization INFO: Task 1264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1264 try 0 launched on executor multi with executor id 877
parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/529bcb75-2330-471c-ba2b-3e632b97e7be_characterizeImage_947422_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/529bcb75-2330-471c-ba2b-3e632b97e7be_characterizeImage_947422_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/379bb84a-62cb-4716-a94e-a4901afcc656_isr_947422_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/isr/947422/379bb84a-62cb-4716-a94e-a4901afcc656_isr_947422_52.stderr
parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5966f6ff8cc44494196fcb0735c4ea with result from task 463
parsl.dataflow.dflow DEBUG: Task 463 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 91a78581e860dcfa15c7dea5c605e6a0
parsl.dataflow.memoization INFO: Task 464 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 878
parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/cc64c4a8-612e-4d8c-a3ea-ee72126af340_calibrate_947418_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/cc64c4a8-612e-4d8c-a3ea-ee72126af340_calibrate_947418_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/0d840017-7316-4b85-8026-b32f8e6e13d6_characterizeImage_947418_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/0d840017-7316-4b85-8026-b32f8e6e13d6_characterizeImage_947418_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:19 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcf492929a8ba2c3ffad63d716f5b72d with result from task 580
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:20 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 580 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 5824fc21038906a4f1da458cf963339f
parsl.dataflow.memoization INFO: Task 581 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 879
parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9975fec-0fbe-47aa-93f2-fcd9fd419228_calibrate_947420_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9975fec-0fbe-47aa-93f2-fcd9fd419228_calibrate_947420_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2d184a87-f8f9-4920-8798-183701062ae3_characterizeImage_947420_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2d184a87-f8f9-4920-8798-183701062ae3_characterizeImage_947420_55.stderr
parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f1f617c955212c09a89ff7495844399a with result from task 304
parsl.dataflow.dflow DEBUG: Task 304 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 24f19b5ebf6e25206f8671f9e2f8a8a1
parsl.dataflow.memoization INFO: Task 305 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 880
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c8b5fd74-be4f-4420-a054-add026bd90dc_calibrate_947523_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c8b5fd74-be4f-4420-a054-add026bd90dc_calibrate_947523_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3c14d3eb-0adf-42fb-ba17-01de2938376f_characterizeImage_947523_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3c14d3eb-0adf-42fb-ba17-01de2938376f_characterizeImage_947523_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:20 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:21 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:21 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a77119c26b3c069b2aaacf01ee34b912 with result from task 1069
parsl.dataflow.dflow DEBUG: Task 1069 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 55405181dca56b6301ee48d6fd47ad4c
parsl.dataflow.memoization INFO: Task 1070 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 881
parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/378ad62e-b585-427a-b4b7-1362db032b8f_calibrate_947420_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/378ad62e-b585-427a-b4b7-1362db032b8f_calibrate_947420_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/5a63493c-1b05-4c53-b781-975d63d04235_characterizeImage_947420_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/5a63493c-1b05-4c53-b781-975d63d04235_characterizeImage_947420_49.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2acffa80cb895c3c71708959d2b4763d with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 6220c97792e1dd75f6d10f06e9f4323c
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 882
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/85fb9679-740e-4b0c-bf2d-1c3e0343ba05_calibrate_947523_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/85fb9679-740e-4b0c-bf2d-1c3e0343ba05_calibrate_947523_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/20004bf9-e33f-4fc5-aaa0-c600df3e0e0d_characterizeImage_947523_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/20004bf9-e33f-4fc5-aaa0-c600df3e0e0d_characterizeImage_947523_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:22 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:22 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ffab501d781a9e638ae00420bc43b192 with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 2bafb49296a8ddf3dd3faf2fdce2be8a
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 883
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9202426-db5d-49ef-b961-037ed86f567f_calibrate_947420_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9202426-db5d-49ef-b961-037ed86f567f_calibrate_947420_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9d68d909-5f05-490f-8403-50028819c5aa_characterizeImage_947420_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9d68d909-5f05-490f-8403-50028819c5aa_characterizeImage_947420_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:23 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:23 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 429
parsl.dataflow.strategy DEBUG: Executor multi has 429 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:27:23 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:27:23 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcf936070c007b64857775d4a6c48396 with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 41edc7ddc6702132372095425311638d
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 884
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ed1af37d-6757-4a41-99c2-a46092180131_calibrate_947420_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ed1af37d-6757-4a41-99c2-a46092180131_calibrate_947420_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/09245fe7-31cf-48d2-a0d5-62490e1336a2_characterizeImage_947420_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/09245fe7-31cf-48d2-a0d5-62490e1336a2_characterizeImage_947420_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:24 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:24 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 553 try 1 failed
parsl.dataflow.dflow ERROR: Task 553 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c2ededd99d6d8fd90be60326e2e9e35 with result from task 553
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 554 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 554 try 0 failed
parsl.dataflow.dflow INFO: Task 554 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1a7390aa-a243-486b-afb7-0ecde7b9ede1_calibrate_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1a7390aa-a243-486b-afb7-0ecde7b9ede1_calibrate_947524_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c6cbb892-3bc1-46c2-a4d0-f4f77275a99a_characterizeImage_947524_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:25 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:25 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd6cb09c73e86d79abb33e2e2694da8c with result from task 1297
parsl.dataflow.dflow DEBUG: Task 1297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1298 has memoization hash db784fa5c7c43afbb142ab6511f828fc
parsl.dataflow.memoization INFO: Task 1298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1298 try 0 launched on executor multi with executor id 885
parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/88364500-088b-4f0e-91d9-5879e2739c6f_calibrate_947523_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/88364500-088b-4f0e-91d9-5879e2739c6f_calibrate_947523_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/691293ff-1a55-490c-ab10-49bfea162f0f_characterizeImage_947523_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/691293ff-1a55-490c-ab10-49bfea162f0f_characterizeImage_947523_43.stderr
parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6580879b48fe8c04c914e6ad601ac836 with result from task 727
parsl.dataflow.dflow DEBUG: Task 727 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 728 has memoization hash e4d57051018bcf6a2de17b7d068c4a69
parsl.dataflow.memoization INFO: Task 728 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 886
parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e429858f-b901-4c29-8700-0da3ba1a3a82_calibrate_947418_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e429858f-b901-4c29-8700-0da3ba1a3a82_calibrate_947418_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d63cd63a-912f-4385-92c4-f5a510c9f88a_characterizeImage_947418_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d63cd63a-912f-4385-92c4-f5a510c9f88a_characterizeImage_947418_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:26 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3dabb34167ba7b9f1304f5379f154a5a with result from task 451
parsl.dataflow.dflow DEBUG: Task 451 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 5cd69b109d7ddbc78aeface14b35804a
parsl.dataflow.memoization INFO: Task 452 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 887
parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9a2be125-367a-4b33-9b4c-fdfbb1fb0812_calibrate_947420_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9a2be125-367a-4b33-9b4c-fdfbb1fb0812_calibrate_947420_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a5ac77f6-c243-4a16-80bf-6049d8759f92_characterizeImage_947420_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a5ac77f6-c243-4a16-80bf-6049d8759f92_characterizeImage_947420_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:26 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95a12943d400bd338eb34d2e3438e0d0 with result from task 49
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 63bd681e5e343371b1ebe28aadc3bdd7
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 888
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ffe281bc-8be1-4cc4-8ef9-09c0c4a772de_calibrate_947420_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ffe281bc-8be1-4cc4-8ef9-09c0c4a772de_calibrate_947420_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/78612383-bc15-4f35-9c57-b580d53b6b0a_characterizeImage_947420_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/78612383-bc15-4f35-9c57-b580d53b6b0a_characterizeImage_947420_27.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 448 try 0 failed
parsl.dataflow.dflow INFO: Task 448 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 4ca186ada87c983c95b26e9453509ccf
parsl.dataflow.memoization INFO: Task 448 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 448 try 1 launched on executor multi with executor id 889
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stderr
parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf33cf693bde9c1deb92d4b851fddd0e with result from task 589
parsl.dataflow.dflow DEBUG: Task 589 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 590 has memoization hash f914d928e124df05103c576d1bf7dadf
parsl.dataflow.memoization INFO: Task 590 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 890
parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/795e3613-c0bb-4e76-8d40-0582a7694b2d_calibrate_947422_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/795e3613-c0bb-4e76-8d40-0582a7694b2d_calibrate_947422_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/524fab00-7aa7-4f0b-ae56-296bdc09bcff_characterizeImage_947422_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/524fab00-7aa7-4f0b-ae56-296bdc09bcff_characterizeImage_947422_47.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 121 try 1 failed
parsl.dataflow.dflow ERROR: Task 121 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 623cc275c57ff512fdc2fb588f16e794 with result from task 121
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 122 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 122 try 0 failed
parsl.dataflow.dflow INFO: Task 122 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d34a3375-4b2d-4096-830a-ddfc4a167b02_calibrate_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d34a3375-4b2d-4096-830a-ddfc4a167b02_calibrate_947418_2.stderr
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7c2537b1-ff9a-478d-a7e9-02fd0bef7c16_characterizeImage_947418_2.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:27 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:27 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:28 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:28 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:28 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:29 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b570ffaa9231a348da7c0853f33d2adf with result from task 283
parsl.dataflow.dflow DEBUG: Task 283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 284 has memoization hash f836fe6ba9a0f68d4402433f77caa4eb
parsl.dataflow.memoization INFO: Task 284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 891
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c5122035-ea9e-4f8f-8002-3f44630cebd2_calibrate_947418_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c5122035-ea9e-4f8f-8002-3f44630cebd2_calibrate_947418_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b0429d5-d0d2-4681-b4b6-607b6e889b58_characterizeImage_947418_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b0429d5-d0d2-4681-b4b6-607b6e889b58_characterizeImage_947418_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:29 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da5af6ec86d42807fda4c81e1a0b00fd with result from task 388
parsl.dataflow.dflow DEBUG: Task 388 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 389 has memoization hash ab1ebe0d3f7d631800bceb781f03e8f0
parsl.dataflow.memoization INFO: Task 389 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 892
parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/acde92fd-7067-4638-9f8e-c7e663ba63c3_calibrate_947420_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/acde92fd-7067-4638-9f8e-c7e663ba63c3_calibrate_947420_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7c99a992-3049-41a4-b255-93cfee256cd3_characterizeImage_947420_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7c99a992-3049-41a4-b255-93cfee256cd3_characterizeImage_947420_42.stderr
parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc8ecf38cee0dece62918cc0863cf862 with result from task 1048
parsl.dataflow.dflow DEBUG: Task 1048 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 45f255c998a6222252a4509846b51c4c
parsl.dataflow.memoization INFO: Task 1049 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 893
parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/0c7ea207-7e26-4b39-89bb-de908d7ddc41_calibrate_947420_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/0c7ea207-7e26-4b39-89bb-de908d7ddc41_calibrate_947420_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057064e4-780b-4927-ab2d-74252e043754_characterizeImage_947420_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057064e4-780b-4927-ab2d-74252e043754_characterizeImage_947420_57.stderr
parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86382ecc666890200c4766a0273ed8fe with result from task 319
parsl.dataflow.dflow DEBUG: Task 319 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 8ffae205ec443f89cda6ea0db765051f
parsl.dataflow.memoization INFO: Task 320 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 894
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2b284254-e29c-48db-ab88-b9898627de91_calibrate_947420_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2b284254-e29c-48db-ab88-b9898627de91_calibrate_947420_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057ad9f7-7420-4595-babe-f7c1853ff502_characterizeImage_947420_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/057ad9f7-7420-4595-babe-f7c1853ff502_characterizeImage_947420_62.stderr
parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d59ade1db57b55feb186d2ab89d3c162 with result from task 718
parsl.dataflow.dflow DEBUG: Task 718 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 719 has memoization hash d0f19da7b54a25e0d4fb3b31733e88c5
parsl.dataflow.memoization INFO: Task 719 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 895
parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/958334ad-80fc-48a2-b930-2bf643272bb1_calibrate_947420_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/958334ad-80fc-48a2-b930-2bf643272bb1_calibrate_947420_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/891e1966-fd80-4ca1-a618-dbc7cea903f2_characterizeImage_947420_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/891e1966-fd80-4ca1-a618-dbc7cea903f2_characterizeImage_947420_15.stderr
parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9008014f9959d0317d53893946cc87f2 with result from task 202
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:29 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow DEBUG: Task 202 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 4c62e7af0521b61c72906addb7f3077f
parsl.dataflow.memoization INFO: Task 203 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 896
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/3ec6246c-e7bb-4bea-99cc-4287ca279156_calibrate_947422_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/3ec6246c-e7bb-4bea-99cc-4287ca279156_calibrate_947422_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb2a838b-c0b7-4f89-970d-d14a93308481_characterizeImage_947422_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb2a838b-c0b7-4f89-970d-d14a93308481_characterizeImage_947422_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:29 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:29 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 005332694cc1a6224320071526c34715 with result from task 322
parsl.dataflow.dflow DEBUG: Task 322 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 38fa5ac9a2842e47d25dd0893eb3260e
parsl.dataflow.memoization INFO: Task 323 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 897
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e64345f-43dd-40fd-b85a-9aca5ddaf937_calibrate_947420_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e64345f-43dd-40fd-b85a-9aca5ddaf937_calibrate_947420_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9a75a310-4d85-49b4-8532-edb4991df967_characterizeImage_947420_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9a75a310-4d85-49b4-8532-edb4991df967_characterizeImage_947420_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51f634817e48dded9ca043615ba7615d with result from task 490
parsl.dataflow.dflow DEBUG: Task 490 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 226dbec23957610548b809da19d225e4
parsl.dataflow.memoization INFO: Task 491 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 898
parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/28e16602-4adc-4a00-9aed-b1ab851620b6_calibrate_947523_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/28e16602-4adc-4a00-9aed-b1ab851620b6_calibrate_947523_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a781f214-5241-47ae-a078-c5a8fbbca4d2_characterizeImage_947523_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a781f214-5241-47ae-a078-c5a8fbbca4d2_characterizeImage_947523_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:30 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 987990a8f3c11e084c9eae7c88ffeead with result from task 370
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow DEBUG: Task 370 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 371 has memoization hash c0729ff4952a5b56e71da3188fac0508
parsl.dataflow.memoization INFO: Task 371 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 899
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d0c4c75e-16c5-46e5-bfe2-d4b103f9fe19_calibrate_947420_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d0c4c75e-16c5-46e5-bfe2-d4b103f9fe19_calibrate_947420_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/10f2c386-3812-47ed-b3a1-3089a8c00f61_characterizeImage_947420_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/10f2c386-3812-47ed-b3a1-3089a8c00f61_characterizeImage_947420_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d6c7c820b5d66aaff86505dd7fbb00e with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash b3856175ad119786a83697c1093512d4
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 900
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6cdccdd9-bc47-4ca7-8984-5fae969e5688_calibrate_947523_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6cdccdd9-bc47-4ca7-8984-5fae969e5688_calibrate_947523_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6fe130c-9256-4300-9cd4-b274e26530fa_characterizeImage_947523_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6fe130c-9256-4300-9cd4-b274e26530fa_characterizeImage_947523_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:31 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1198 try 0 failed
parsl.dataflow.dflow INFO: Task 1198 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1198 has memoization hash 5fa975ac91ec5bdd4f76d628699a600a
parsl.dataflow.memoization INFO: Task 1198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1198 try 1 launched on executor multi with executor id 901
parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dff5cf7fa99bcadc5bee4b1bfd7bbf73 with result from task 676
parsl.dataflow.dflow DEBUG: Task 676 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 17d1fd476a55139c5b44860463d7a9d7
parsl.dataflow.memoization INFO: Task 677 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 902
parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ed589bc3-bda7-4745-a3c9-f3739d8e42c7_calibrate_947418_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ed589bc3-bda7-4745-a3c9-f3739d8e42c7_calibrate_947418_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cea84278-0920-4507-ab1c-d71d2be72094_characterizeImage_947418_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cea84278-0920-4507-ab1c-d71d2be72094_characterizeImage_947418_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6ec749f479cadd5645b4e7a476ba3cc with result from task 421
parsl.dataflow.dflow DEBUG: Task 421 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 6c6d582d7fd9603676bff82be674b044
parsl.dataflow.memoization INFO: Task 422 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 903
parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/33daa05f-4304-42fa-85e1-4fe92f609572_calibrate_947418_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/33daa05f-4304-42fa-85e1-4fe92f609572_calibrate_947418_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/23c1c262-ea5b-40f1-844b-74def6b95e13_characterizeImage_947418_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/23c1c262-ea5b-40f1-844b-74def6b95e13_characterizeImage_947418_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:32 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:33 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:33 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f971ca105259cea937a6131bb5cfc78f with result from task 349
parsl.dataflow.dflow DEBUG: Task 349 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 7c390f7e3309306653bf2784b435ed92
parsl.dataflow.memoization INFO: Task 350 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 904
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5dc8e937-6142-4ff5-a460-1d512288f429_calibrate_947418_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5dc8e937-6142-4ff5-a460-1d512288f429_calibrate_947418_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9d9a7211-d59b-484b-81a5-849898877047_characterizeImage_947418_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9d9a7211-d59b-484b-81a5-849898877047_characterizeImage_947418_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:33 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:33 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0723951560651cf84d35e422a963bd96 with result from task 487
parsl.dataflow.dflow DEBUG: Task 487 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 99ac2eb1e4149f21337f662d748198ed
parsl.dataflow.memoization INFO: Task 488 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 905
parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e89a2aad-cb28-4ec9-b7e8-7c1dce5a5c49_calibrate_947422_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e89a2aad-cb28-4ec9-b7e8-7c1dce5a5c49_calibrate_947422_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebb751ff-fadd-47c6-bf73-bdbe34f81d86_characterizeImage_947422_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebb751ff-fadd-47c6-bf73-bdbe34f81d86_characterizeImage_947422_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:33 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:33 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74964db2664efd20f188564134a82f85 with result from task 325
parsl.dataflow.dflow DEBUG: Task 325 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 41dda9a88216271970cc70766a813114
parsl.dataflow.memoization INFO: Task 326 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 906
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/d93f4e49-c81b-4206-bb3b-0e89720a21d1_calibrate_947523_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/d93f4e49-c81b-4206-bb3b-0e89720a21d1_calibrate_947523_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/aa2ea0c8-68b1-4ec7-b179-a946b1d3e0e0_characterizeImage_947523_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/aa2ea0c8-68b1-4ec7-b179-a946b1d3e0e0_characterizeImage_947523_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:34 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 69697bdaaf51446de99ea29b26ac8bfa with result from task 427
parsl.dataflow.dflow DEBUG: Task 427 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 428 has memoization hash dc6f53b6c01809bc99896b6bfef95e1b
parsl.dataflow.memoization INFO: Task 428 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f525c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 907
parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ea8df07f-230f-481a-8fec-0c90db5f1885_calibrate_947420_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ea8df07f-230f-481a-8fec-0c90db5f1885_calibrate_947420_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/cd3408c0-bcf7-4bd1-955e-6c4b7afcf3ff_characterizeImage_947420_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/cd3408c0-bcf7-4bd1-955e-6c4b7afcf3ff_characterizeImage_947420_3.stderr
parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eac5d0679f44d796630590cd316615ee with result from task 664
parsl.dataflow.dflow DEBUG: Task 664 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 91bb7a2fe45a58ea8573b0840c81e077
parsl.dataflow.memoization INFO: Task 665 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 908
parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e411f766-31e9-42c3-a130-78d2f855afa0_calibrate_947422_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e411f766-31e9-42c3-a130-78d2f855afa0_calibrate_947422_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e473f9fd-3fc4-4d5d-b10a-486b93ade134_characterizeImage_947422_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e473f9fd-3fc4-4d5d-b10a-486b93ade134_characterizeImage_947422_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cde30515c2f5f85a6092c957fc8b1172 with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 2d7feed9d71f93cc6b524b97c0812254
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 909
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a3b2731-984a-4ec8-9a85-77efa0423a0c_calibrate_947422_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a3b2731-984a-4ec8-9a85-77efa0423a0c_calibrate_947422_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0fb71db5-9272-4dff-a8d0-c0d8de04726e_characterizeImage_947422_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0fb71db5-9272-4dff-a8d0-c0d8de04726e_characterizeImage_947422_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b49253f40f78dd8587e7b9fa12851ede with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 8f98c7437e74fe0c4c39690e3b27d7c4
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 910
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/18241bb4-40e0-4a1f-a618-8f915ded38fc_calibrate_947418_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/18241bb4-40e0-4a1f-a618-8f915ded38fc_calibrate_947418_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c0df162f-8f7d-4732-8e10-58a5cbbde875_characterizeImage_947418_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c0df162f-8f7d-4732-8e10-58a5cbbde875_characterizeImage_947418_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:35 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db1983a908bf28b81bbe6147a4c07ef1 with result from task 481
parsl.dataflow.dflow DEBUG: Task 481 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 482 has memoization hash db05ccb68de03f8c23419eb93c129d9a
parsl.dataflow.memoization INFO: Task 482 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 911
parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0586ddad-0baa-4ea7-972b-672e264dd6cf_calibrate_947418_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0586ddad-0baa-4ea7-972b-672e264dd6cf_calibrate_947418_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8a1436da-1f38-488a-895c-dfdf4ee89ec9_characterizeImage_947418_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8a1436da-1f38-488a-895c-dfdf4ee89ec9_characterizeImage_947418_60.stderr
parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5519ed4dd97ccb375ebf1f1019093fcb with result from task 907
parsl.dataflow.dflow DEBUG: Task 907 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 1ba1b5b8cef8c69f999586a501360f92
parsl.dataflow.memoization INFO: Task 908 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 912
parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b0e9cb50-43e9-4dd3-8eb7-5e181d5d7f54_calibrate_947523_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b0e9cb50-43e9-4dd3-8eb7-5e181d5d7f54_calibrate_947523_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9dd1babb-c795-476d-89a4-8b118dd4d222_characterizeImage_947523_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9dd1babb-c795-476d-89a4-8b118dd4d222_characterizeImage_947523_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:36 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53d61f5f65d75c7e6f41998af69234ce with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash a006a6f62e80c5636837fe5bef55948f
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 913
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/385130ac-5b5a-431e-a51a-8f305db698f0_calibrate_947422_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/385130ac-5b5a-431e-a51a-8f305db698f0_calibrate_947422_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb4a7d5c-8f33-409e-b14a-2b2b1bddfcbf_characterizeImage_947422_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/bb4a7d5c-8f33-409e-b14a-2b2b1bddfcbf_characterizeImage_947422_33.stderr
parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff7508b26ccee4df03d5caa6240e1fd2 with result from task 595
parsl.dataflow.dflow DEBUG: Task 595 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 1d4669c1cf9065011d5430679c69db30
parsl.dataflow.memoization INFO: Task 596 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 914
parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8d6731a2-1877-42de-9c5d-511e06a55711_calibrate_947422_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8d6731a2-1877-42de-9c5d-511e06a55711_calibrate_947422_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/430c2d69-5983-4670-b26d-bf4013902e42_characterizeImage_947422_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/430c2d69-5983-4670-b26d-bf4013902e42_characterizeImage_947422_53.stderr
parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32987da6a0d573e147b3b1750200665d with result from task 622
parsl.dataflow.dflow DEBUG: Task 622 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 06d00f857c723ccb1f300de7cf2bbee2
parsl.dataflow.memoization INFO: Task 623 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 915
parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e8a938c5-d8b1-48bc-8f93-919da3c65898_calibrate_947523_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e8a938c5-d8b1-48bc-8f93-919da3c65898_calibrate_947523_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fb4d9c2e-e8df-4160-8c5a-98668802a21c_characterizeImage_947523_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fb4d9c2e-e8df-4160-8c5a-98668802a21c_characterizeImage_947523_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:36 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3905bc78c6323720653dd62c4ee08ee0 with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash c6138c27d96bcfa373ecd9eed64774c9
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 916
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/83cf5273-d236-4c2d-aff5-4069d069f921_calibrate_947524_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/83cf5273-d236-4c2d-aff5-4069d069f921_calibrate_947524_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/42830f79-c4c7-4845-b53c-d74e2bed84c9_characterizeImage_947524_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/42830f79-c4c7-4845-b53c-d74e2bed84c9_characterizeImage_947524_57.stderr
parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16dc9a0de465a05712a4ca0e53e1d4c8 with result from task 877
parsl.dataflow.dflow DEBUG: Task 877 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 59d895eb15b5b61e606d481475d5493a
parsl.dataflow.memoization INFO: Task 878 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 917
parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f3ac932e-321a-4568-9312-8c4e38493000_calibrate_947418_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f3ac932e-321a-4568-9312-8c4e38493000_calibrate_947418_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b941cd9-91b6-4bf6-a2f8-1a4e99a1d7f4_characterizeImage_947418_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5b941cd9-91b6-4bf6-a2f8-1a4e99a1d7f4_characterizeImage_947418_33.stderr
parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc754335e7bd384afe6659c940f9e0c with result from task 232
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:36 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 47f28732d96b69964ad25a24220af972
parsl.dataflow.memoization INFO: Task 233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 918
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2d2c706f-135b-42d5-9c0b-84536c68b73d_calibrate_947420_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2d2c706f-135b-42d5-9c0b-84536c68b73d_calibrate_947420_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2bbdec7a-a2eb-4f34-9e46-42f07006d808_characterizeImage_947420_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2bbdec7a-a2eb-4f34-9e46-42f07006d808_characterizeImage_947420_36.stderr
parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72c4189881b5d6540b267e721bf9e8db with result from task 289
parsl.dataflow.dflow DEBUG: Task 289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 6d8885b4cec274fad242485ed83b06ff
parsl.dataflow.memoization INFO: Task 290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 919
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/a3ea8622-e4a0-444e-b570-00c68bc82a69_calibrate_947420_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/a3ea8622-e4a0-444e-b570-00c68bc82a69_calibrate_947420_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be4ec842-adce-4bfa-8a04-b4dee174d0dc_characterizeImage_947420_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be4ec842-adce-4bfa-8a04-b4dee174d0dc_characterizeImage_947420_43.stderr
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbbbf1a15d48d07b31eb9e45a4cdbe9 with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 7576c0faf337d35a7c231a1813cc46f1
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 920
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/772b2184-dfe5-45c4-acb9-2d2b3980310a_calibrate_947524_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/772b2184-dfe5-45c4-acb9-2d2b3980310a_calibrate_947524_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/e2c156d6-acbf-484c-a19e-1942d7d0ada2_characterizeImage_947524_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/e2c156d6-acbf-484c-a19e-1942d7d0ada2_characterizeImage_947524_23.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cbfe4eee516760fccf42f47f5d75a0b2 with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash afd5f3c3ec3e83324c73cbaef6d7117b
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 921
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/831e3521-2152-483d-b80e-cea8639ea31c_calibrate_947524_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/831e3521-2152-483d-b80e-cea8639ea31c_calibrate_947524_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/764d1136-bb72-42df-bc7c-66de51da84b1_characterizeImage_947524_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/764d1136-bb72-42df-bc7c-66de51da84b1_characterizeImage_947524_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:37 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3f9369c7b3ff175fe093c2a913e4b5f with result from task 715
parsl.dataflow.dflow DEBUG: Task 715 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 3d6dc6a0a62481388f1b8b5e71e63820
parsl.dataflow.memoization INFO: Task 716 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 922
parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0dda74e6-b3a9-4f6c-ac86-c3131c013fcc_calibrate_947523_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0dda74e6-b3a9-4f6c-ac86-c3131c013fcc_calibrate_947523_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9d5a5885-339d-47d2-8d41-5040bd39ec46_characterizeImage_947523_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/9d5a5885-339d-47d2-8d41-5040bd39ec46_characterizeImage_947523_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:37 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:38 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2b9e610a639eeb4b37cc565282cd8bb with result from task 814
parsl.dataflow.dflow DEBUG: Task 814 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 5468a086b7bf2beb9931456f75050c15
parsl.dataflow.memoization INFO: Task 815 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 923
parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/bd0c5fa9-8188-4554-97db-ac29c3dea1bb_calibrate_947418_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/bd0c5fa9-8188-4554-97db-ac29c3dea1bb_calibrate_947418_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6cdb79a0-0e21-4787-8e6c-e6d8168f1964_characterizeImage_947418_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6cdb79a0-0e21-4787-8e6c-e6d8168f1964_characterizeImage_947418_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:38 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:38 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:39 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:39 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:40 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5de0a64f42183bdc41c2f02ed1e5f6c6 with result from task 454
parsl.dataflow.dflow DEBUG: Task 454 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 7b0c0e0ff62918856ea456698b4cd6ab
parsl.dataflow.memoization INFO: Task 455 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 924
parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f54d2f6-74ff-4640-b343-cef309f18b16_calibrate_947523_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f54d2f6-74ff-4640-b343-cef309f18b16_calibrate_947523_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4972949c-a6f2-4f1c-bfa5-96f74d2c73f9_characterizeImage_947523_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4972949c-a6f2-4f1c-bfa5-96f74d2c73f9_characterizeImage_947523_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:40 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f1ddbd2a2de7153fb70729694a168be with result from task 244
parsl.dataflow.dflow DEBUG: Task 244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 22390e0e26e01984fc4d9bc7489a0278
parsl.dataflow.memoization INFO: Task 245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 925
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5b7ed98a-fc65-4057-b963-e6689d20c24f_calibrate_947422_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5b7ed98a-fc65-4057-b963-e6689d20c24f_calibrate_947422_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a2536188-88f4-4baf-8e21-da13ff19883e_characterizeImage_947422_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a2536188-88f4-4baf-8e21-da13ff19883e_characterizeImage_947422_36.stderr
parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8ff1e95a63d5b1e044dace157b107c9 with result from task 241
parsl.dataflow.dflow DEBUG: Task 241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 46e6a47c8b1d50202aaa99fff5fb2645
parsl.dataflow.memoization INFO: Task 242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 926
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/599f8675-1617-45d5-b4a1-c5e6ae0ecf31_calibrate_947420_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/599f8675-1617-45d5-b4a1-c5e6ae0ecf31_calibrate_947420_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b54614e5-1f5d-48bc-9bcf-908696ad9ee1_characterizeImage_947420_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b54614e5-1f5d-48bc-9bcf-908696ad9ee1_characterizeImage_947420_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:40 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9702670b5282678251a3ef532036e777 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash cca61043a9e7b41715b3ff48b3a1f7b5
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 927
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8840cf13-6f08-4beb-8781-47ff36837dc0_calibrate_947420_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8840cf13-6f08-4beb-8781-47ff36837dc0_calibrate_947420_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9820409f-45aa-4154-a3a3-0fa61ced5422_characterizeImage_947420_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9820409f-45aa-4154-a3a3-0fa61ced5422_characterizeImage_947420_21.stderr
parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6a2769da80966cade315178a1e7ec9 with result from task 436
parsl.dataflow.dflow DEBUG: Task 436 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 241335ebd86fab942aae5782f2177628
parsl.dataflow.memoization INFO: Task 437 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 928
parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/29deaaaa-2eee-48d9-8ecd-331598e38e59_calibrate_947524_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/29deaaaa-2eee-48d9-8ecd-331598e38e59_calibrate_947524_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8cb46067-cb45-4770-83ee-ffbc539481ee_characterizeImage_947524_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8cb46067-cb45-4770-83ee-ffbc539481ee_characterizeImage_947524_15.stderr
parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2bfbf969e94699fd1e6cbf78a867547c with result from task 328
parsl.dataflow.dflow DEBUG: Task 328 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 10243591bb23f58a3f07330792bd1c1c
parsl.dataflow.memoization INFO: Task 329 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f03f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 929
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ded2a22e-0dce-4109-b7c9-36c8fb78ac51_calibrate_947418_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ded2a22e-0dce-4109-b7c9-36c8fb78ac51_calibrate_947418_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/76d545f7-4666-4e62-8134-8f79449df4d5_characterizeImage_947418_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/76d545f7-4666-4e62-8134-8f79449df4d5_characterizeImage_947418_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:41 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2b33e1abc14cba1bb337e451e585a0f with result from task 796
parsl.dataflow.dflow DEBUG: Task 796 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 797 has memoization hash aa4f3655db55378b3c4e9c93819b2d73
parsl.dataflow.memoization INFO: Task 797 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 930
parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e464522-6c1d-4c8e-9499-be303508e0a7_calibrate_947420_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e464522-6c1d-4c8e-9499-be303508e0a7_calibrate_947420_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a0a77c70-b8b5-4245-a950-be787fcb9f95_characterizeImage_947420_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a0a77c70-b8b5-4245-a950-be787fcb9f95_characterizeImage_947420_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:41 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f884b0e9f7b2414e10a4fb88db8fa343 with result from task 910
parsl.dataflow.dflow DEBUG: Task 910 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 911 has memoization hash c65c0b887f9c76c85cbff1e631870faf
parsl.dataflow.memoization INFO: Task 911 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 931
parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/32dd0021-2ea0-49e8-9818-9cd56f3ad929_calibrate_947422_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/32dd0021-2ea0-49e8-9818-9cd56f3ad929_calibrate_947422_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1dcc9b22-d583-48f4-aabb-247f6b6ae438_characterizeImage_947422_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1dcc9b22-d583-48f4-aabb-247f6b6ae438_characterizeImage_947422_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:41 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:42 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09cd6f07cceb67bbd49ce16cc0411010 with result from task 790
parsl.dataflow.dflow DEBUG: Task 790 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 701668ddbc648bb8276e61c9b78a1c58
parsl.dataflow.memoization INFO: Task 791 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 932
parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/88c6a893-693c-47d8-800f-487dcf993011_calibrate_947422_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/88c6a893-693c-47d8-800f-487dcf993011_calibrate_947422_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/656be7ee-cac3-41ce-afdf-39177977a93b_characterizeImage_947422_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/656be7ee-cac3-41ce-afdf-39177977a93b_characterizeImage_947422_24.stderr
parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3af06f42209a7112bf486d4514834fc0 with result from task 784
parsl.dataflow.dflow DEBUG: Task 784 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 1283dc738ac86b881f94a11cc0eb304b
parsl.dataflow.memoization INFO: Task 785 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 933
parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/8fe99e24-71e0-4cf7-a51d-4a90d5fc51f9_calibrate_947523_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/8fe99e24-71e0-4cf7-a51d-4a90d5fc51f9_calibrate_947523_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fbc98c96-7f9d-4a03-98a7-a20f660266b3_characterizeImage_947523_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/fbc98c96-7f9d-4a03-98a7-a20f660266b3_characterizeImage_947523_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:42 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c72a43c1090f7146c77c20b393931bce with result from task 457
parsl.dataflow.dflow DEBUG: Task 457 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 2355a907ef7d78480bbb7a95f40d4a5e
parsl.dataflow.memoization INFO: Task 458 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 934
parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/62cc8b74-454f-4d70-951d-ab8480c71027_calibrate_947422_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/62cc8b74-454f-4d70-951d-ab8480c71027_calibrate_947422_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2e30590f-4f5b-4fae-8a2a-643e03aec06e_characterizeImage_947422_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2e30590f-4f5b-4fae-8a2a-643e03aec06e_characterizeImage_947422_10.stderr
parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d1d231a55bced3b041d59decbdefeca with result from task 541
parsl.dataflow.dflow DEBUG: Task 541 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 5f41f69d54f35414c51bdb3f7cb53408
parsl.dataflow.memoization INFO: Task 542 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 935
parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fce1d0e5-d9af-4944-ba03-69006126e684_calibrate_947418_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fce1d0e5-d9af-4944-ba03-69006126e684_calibrate_947418_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/209cd2db-ac72-4d51-9f4e-e415b59456e9_characterizeImage_947418_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/209cd2db-ac72-4d51-9f4e-e415b59456e9_characterizeImage_947418_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:42 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72eab86fb06a9928680384f38ec552f6 with result from task 838
parsl.dataflow.dflow DEBUG: Task 838 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 839 has memoization hash aca31465f37ca2dfc2ba70cbda031fae
parsl.dataflow.memoization INFO: Task 839 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 936
parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4a5ddfc7-31a6-4f23-90a4-f4cb715dd209_calibrate_947420_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4a5ddfc7-31a6-4f23-90a4-f4cb715dd209_calibrate_947420_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/77d4440d-ed41-4625-9263-621bd0f45f0a_characterizeImage_947420_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/77d4440d-ed41-4625-9263-621bd0f45f0a_characterizeImage_947420_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:43 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 145008bc2011f379b9128ef1ea1421a1 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 2d3af897781808f4b5e2d80e3bbaed56
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 937
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/53c31ddc-3112-4557-bb44-a31463ac05b0_calibrate_947418_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/53c31ddc-3112-4557-bb44-a31463ac05b0_calibrate_947418_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2c9a6f8f-8fbf-4f82-b091-56677244b580_characterizeImage_947418_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2c9a6f8f-8fbf-4f82-b091-56677244b580_characterizeImage_947418_4.stderr
parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34f0c061a94b7de311bd61c0e7b60366 with result from task 565
parsl.dataflow.dflow DEBUG: Task 565 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 566 has memoization hash c84c89dd6ae018171e9fd4962f8c3064
parsl.dataflow.memoization INFO: Task 566 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 938
parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/508252aa-a691-41bc-8900-181305df0eb9_calibrate_947526_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/508252aa-a691-41bc-8900-181305df0eb9_calibrate_947526_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bae9f807-679f-4a81-b1f7-6403fffb733d_characterizeImage_947526_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bae9f807-679f-4a81-b1f7-6403fffb733d_characterizeImage_947526_59.stderr
parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8adbb92919af5f34792512264373dad with result from task 358
parsl.dataflow.dflow DEBUG: Task 358 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 9ec4073920be06485c3a2ef9607beaa6
parsl.dataflow.memoization INFO: Task 359 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 939
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0aef1900-0b49-4939-a5cd-e1cd5a10f446_calibrate_947523_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0aef1900-0b49-4939-a5cd-e1cd5a10f446_calibrate_947523_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/1d6ea37f-bf3f-401d-a72e-beb25614b886_characterizeImage_947523_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/1d6ea37f-bf3f-401d-a72e-beb25614b886_characterizeImage_947523_42.stderr
parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bac6812e9f8bd84e39d98b772902be02 with result from task 544
parsl.dataflow.dflow DEBUG: Task 544 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 545 has memoization hash dba0f1032da4ff63056930f9b04cd2c6
parsl.dataflow.memoization INFO: Task 545 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 940
parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0e2dc453-2881-4d38-9b38-1e4cd7f6a494_calibrate_947418_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0e2dc453-2881-4d38-9b38-1e4cd7f6a494_calibrate_947418_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/3038955f-0527-4148-973c-8ddd4bf3e628_characterizeImage_947418_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/3038955f-0527-4148-973c-8ddd4bf3e628_characterizeImage_947418_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:43 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 491c2c2ee149832357064e3d3ed2d00a with result from task 268
parsl.dataflow.dflow DEBUG: Task 268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 6d107adab47c7d77d4f0bb006f9c3b85
parsl.dataflow.memoization INFO: Task 269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 941
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/45f69b97-7827-4b72-8783-1b142aaecf51_calibrate_947528_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/45f69b97-7827-4b72-8783-1b142aaecf51_calibrate_947528_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/efaf004a-6057-417f-8209-91294f59ae47_characterizeImage_947528_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/efaf004a-6057-417f-8209-91294f59ae47_characterizeImage_947528_31.stderr
parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b74634238c16a9193c1281b9a1fdbecc with result from task 802
parsl.dataflow.dflow DEBUG: Task 802 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 2fa18a4609706c98c781d74d9f779fbe
parsl.dataflow.memoization INFO: Task 803 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 942
parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/573a53ab-cd47-47c5-84ca-bd04ef2edf58_calibrate_947524_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/573a53ab-cd47-47c5-84ca-bd04ef2edf58_calibrate_947524_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/86daac64-0c72-497c-877f-bfac49bc9253_characterizeImage_947524_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/86daac64-0c72-497c-877f-bfac49bc9253_characterizeImage_947524_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:43 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 077570f6bbb13416bbffd9ec421a86b3 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash e4e9431a629039b7b30097f391f9297b
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 943
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6762cca3-aa1f-49bb-b2ca-f190238663d3_calibrate_947418_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6762cca3-aa1f-49bb-b2ca-f190238663d3_calibrate_947418_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ce6f248e-9f5f-4cd0-8a1c-7b742e477932_characterizeImage_947418_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ce6f248e-9f5f-4cd0-8a1c-7b742e477932_characterizeImage_947418_14.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1198 try 1 failed
parsl.dataflow.dflow ERROR: Task 1198 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa975ac91ec5bdd4f76d628699a600a with result from task 1198
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1199 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1199 try 0 failed
parsl.dataflow.dflow INFO: Task 1199 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d8ff2852-eee4-40b6-a3f9-eda179e5d953_calibrate_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d8ff2852-eee4-40b6-a3f9-eda179e5d953_calibrate_947418_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eff0a9c2-9669-46ea-8be3-a91b72993278_characterizeImage_947418_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:44 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:44 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e989471fe96c4b1694c198ba66dc3a38 with result from task 577
parsl.dataflow.dflow DEBUG: Task 577 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 1dced2c24f2463085c67dc07bbf23b4d
parsl.dataflow.memoization INFO: Task 578 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dccb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 944
parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2335e514-00e5-4cd0-878b-c05d9c32c3af_calibrate_947523_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2335e514-00e5-4cd0-878b-c05d9c32c3af_calibrate_947523_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7d05f873-6205-468c-9904-559d7ad63e8f_characterizeImage_947523_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7d05f873-6205-468c-9904-559d7ad63e8f_characterizeImage_947523_23.stderr
parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9bf1125ddeabad3f93fd7f927c0e98aa with result from task 406
parsl.dataflow.dflow DEBUG: Task 406 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 0d0f1f7af635d909909fd2b807a1549d
parsl.dataflow.memoization INFO: Task 407 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 945
parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e4a945b2-9a76-4120-a18a-dcdaae3d1021_calibrate_947422_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e4a945b2-9a76-4120-a18a-dcdaae3d1021_calibrate_947422_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0b18095a-87b6-437b-889a-0bea823485ce_characterizeImage_947422_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0b18095a-87b6-437b-889a-0bea823485ce_characterizeImage_947422_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:44 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40810a8d2f736ec28772ab5530a906eb with result from task 478
parsl.dataflow.dflow DEBUG: Task 478 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 5b6e10a99fed9990255e8a0002dd6197
parsl.dataflow.memoization INFO: Task 479 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 946
parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2a58143b-6e1a-429d-933e-8a2cdfd2fd7d_calibrate_947528_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2a58143b-6e1a-429d-933e-8a2cdfd2fd7d_calibrate_947528_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af266b67-207e-49a1-96a3-fe39a6c061c3_characterizeImage_947528_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af266b67-207e-49a1-96a3-fe39a6c061c3_characterizeImage_947528_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:45 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31d26feb67260add5faca529f6684723 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 3f0176b8e0a3b643d9f6d1255f23c044
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 947
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/677a761e-1198-4c4c-a170-f850c107069e_calibrate_947528_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/677a761e-1198-4c4c-a170-f850c107069e_calibrate_947528_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/e43dbd4c-e3ad-45b2-b3a8-559e0cf4be00_characterizeImage_947528_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/e43dbd4c-e3ad-45b2-b3a8-559e0cf4be00_characterizeImage_947528_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:45 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be9e3163f8457a48c4bae57977f365fa with result from task 751
parsl.dataflow.dflow DEBUG: Task 751 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 031bbc2f22c96c623d123b97083cfe7f
parsl.dataflow.memoization INFO: Task 752 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 948
parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f1c852d2-dfc5-4b88-a982-097fbadf3b83_calibrate_947422_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f1c852d2-dfc5-4b88-a982-097fbadf3b83_calibrate_947422_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/205e6d1f-31ff-41e0-90b7-a54c2ee5d559_characterizeImage_947422_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/205e6d1f-31ff-41e0-90b7-a54c2ee5d559_characterizeImage_947422_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:46 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 450bdf9bb809d6a1ea991e0c48a2f801 with result from task 1270
parsl.dataflow.dflow DEBUG: Task 1270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1271 has memoization hash a2fb55bd3082c8214c12a646b84489e9
parsl.dataflow.memoization INFO: Task 1271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1271 try 0 launched on executor multi with executor id 949
parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/68720f84-0af8-4548-a790-96772b0f3554_calibrate_947422_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/68720f84-0af8-4548-a790-96772b0f3554_calibrate_947422_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4775f391-f7ca-4b61-8fda-a66f8e2b77b5_characterizeImage_947422_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4775f391-f7ca-4b61-8fda-a66f8e2b77b5_characterizeImage_947422_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:46 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd1220829bf7493b8e39b16975e2c6a2 with result from task 667
parsl.dataflow.dflow DEBUG: Task 667 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 0038ae9c9eaca57c1376dfbcff45c663
parsl.dataflow.memoization INFO: Task 668 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 950
parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5c82cec6-4658-4943-ac9b-f9ea8eb03b61_calibrate_947420_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5c82cec6-4658-4943-ac9b-f9ea8eb03b61_calibrate_947420_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0db3de36-e22e-40e1-8637-5ae1e1242a00_characterizeImage_947420_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0db3de36-e22e-40e1-8637-5ae1e1242a00_characterizeImage_947420_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:46 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:47 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebddc8d5e319f43b771b24d9f8a2e150 with result from task 721
parsl.dataflow.dflow DEBUG: Task 721 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 722 has memoization hash d161ec43bad3e0845b4f939268d566d8
parsl.dataflow.memoization INFO: Task 722 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 951
parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/53aacb15-efed-44d4-8caf-6a7664f49a55_calibrate_947422_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/53aacb15-efed-44d4-8caf-6a7664f49a55_calibrate_947422_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb62cb70-ad0e-4e82-9ee6-1f6b44cb779f_characterizeImage_947422_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb62cb70-ad0e-4e82-9ee6-1f6b44cb779f_characterizeImage_947422_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:47 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79bd184581f4918019b8a428d9efd47f with result from task 760
parsl.dataflow.dflow DEBUG: Task 760 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 4a330421201195b0ee04512ce3eabcd1
parsl.dataflow.memoization INFO: Task 761 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 952
parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/efb615bb-f462-4dfe-a9f3-00faa2e1a60f_calibrate_947420_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/efb615bb-f462-4dfe-a9f3-00faa2e1a60f_calibrate_947420_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9e2f4441-75a6-4e32-a25a-00d4f9aa5e37_characterizeImage_947420_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9e2f4441-75a6-4e32-a25a-00d4f9aa5e37_characterizeImage_947420_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:48 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:48 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a5bcabccf633bde7a25b4b391e55a9b with result from task 670
parsl.dataflow.dflow DEBUG: Task 670 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 5686900b77c05dc9663a16b0b3baf33a
parsl.dataflow.memoization INFO: Task 671 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 953
parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15c07d76-2af2-48fc-b77a-8b5d1eb4fced_calibrate_947420_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15c07d76-2af2-48fc-b77a-8b5d1eb4fced_calibrate_947420_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ec3d92b3-bb82-4efb-868b-cbaa5960f596_characterizeImage_947420_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ec3d92b3-bb82-4efb-868b-cbaa5960f596_characterizeImage_947420_35.stderr
parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c36053e1771d3d43c41d62e0ddcf7ab with result from task 520
parsl.dataflow.dflow DEBUG: Task 520 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 143bc622b157e625b4c0cb978c8c6be7
parsl.dataflow.memoization INFO: Task 521 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 954
parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ba6404b9-5b95-4a6d-9578-b94d264c873f_calibrate_947523_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ba6404b9-5b95-4a6d-9578-b94d264c873f_calibrate_947523_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/707cfa30-aa44-456a-b682-33707c87cf4c_characterizeImage_947523_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/707cfa30-aa44-456a-b682-33707c87cf4c_characterizeImage_947523_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:48 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13c9cfe299b2d2f727b9834c8b668973 with result from task 493
parsl.dataflow.dflow DEBUG: Task 493 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 494 has memoization hash dc70190ab755949f712ab340224ee97c
parsl.dataflow.memoization INFO: Task 494 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 955
parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c77ff5cc-70d2-47f8-9db8-abcdfa782dd2_calibrate_947418_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c77ff5cc-70d2-47f8-9db8-abcdfa782dd2_calibrate_947418_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ba2e67e2-acbf-458b-b979-bde2a101eada_characterizeImage_947418_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ba2e67e2-acbf-458b-b979-bde2a101eada_characterizeImage_947418_34.stderr
parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bdd46f582065b04fe9a64f7c86c99bf2 with result from task 496
parsl.dataflow.dflow DEBUG: Task 496 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 8072a201c5a74b67c60336c2d6c9f509
parsl.dataflow.memoization INFO: Task 497 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 956
parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9b271258-0352-4981-982c-4277515f9b76_calibrate_947523_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9b271258-0352-4981-982c-4277515f9b76_calibrate_947523_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/feb4bdae-825f-4944-9072-d8a5ed54dc67_characterizeImage_947523_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/feb4bdae-825f-4944-9072-d8a5ed54dc67_characterizeImage_947523_52.stderr
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6eca07188d8751c916a532f01d0289a2 with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 113633036f1d48bdc34ae33e2b8fc67f
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 957
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/956e513c-2a54-4391-8f2e-ce5a4af632be_calibrate_947523_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/956e513c-2a54-4391-8f2e-ce5a4af632be_calibrate_947523_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/0b19434a-e963-4bc1-9e4e-02507c278d0b_characterizeImage_947523_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/0b19434a-e963-4bc1-9e4e-02507c278d0b_characterizeImage_947523_21.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9fff4c642f960bb13e0e697431172e with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 7466c6efd843a7a354f454b3b26a7ecd
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 958
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fe845a2e-3b97-4f19-a6fd-932dffb6fc9b_calibrate_947524_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fe845a2e-3b97-4f19-a6fd-932dffb6fc9b_calibrate_947524_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/da9c948a-73f4-4b52-8201-b985b0819e3f_characterizeImage_947524_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/da9c948a-73f4-4b52-8201-b985b0819e3f_characterizeImage_947524_14.stderr
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da498683e1bdecc64068b9d6a0c28778 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash d64e4509957893fc69b0a3b42e42a6c2
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 959
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f2e4f294-addb-4b9b-b596-2c4c1fb7e0c0_calibrate_947523_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f2e4f294-addb-4b9b-b596-2c4c1fb7e0c0_calibrate_947523_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/cf81a863-2369-4877-adc8-934424f92f23_characterizeImage_947523_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/cf81a863-2369-4877-adc8-934424f92f23_characterizeImage_947523_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:49 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a0a05a34ba63c3de245c65cc9e2c270 with result from task 466
parsl.dataflow.dflow DEBUG: Task 466 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 1b5a133677ece71e360976e70a85e0e4
parsl.dataflow.memoization INFO: Task 467 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 960
parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f697da3-d6be-43f1-aea5-1318f3b201b4_calibrate_947523_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f697da3-d6be-43f1-aea5-1318f3b201b4_calibrate_947523_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2cdfa2bf-e496-4c30-906d-3569c9bac572_characterizeImage_947523_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2cdfa2bf-e496-4c30-906d-3569c9bac572_characterizeImage_947523_45.stderr
parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2086e30432abd52d3d4fd7120000d20 with result from task 811
parsl.dataflow.dflow DEBUG: Task 811 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 812 has memoization hash e61774dec3f5c020996a73ae30bbb1f8
parsl.dataflow.memoization INFO: Task 812 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 961
parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/20a32d75-caf4-4680-bd0d-2dd97d117cc6_calibrate_947418_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/20a32d75-caf4-4680-bd0d-2dd97d117cc6_calibrate_947418_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/39330eef-8750-4e53-9987-6a58ffe5564d_characterizeImage_947418_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/39330eef-8750-4e53-9987-6a58ffe5564d_characterizeImage_947418_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:49 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01db9f60b67a797a655f9e99f4dac8ca with result from task 400
parsl.dataflow.dflow DEBUG: Task 400 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 6c30cd1bdb7e1fe5460908f7b28039ee
parsl.dataflow.memoization INFO: Task 401 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f509a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 962
parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a908ce4-a357-4bd0-bdd6-5e1f76d02f58_calibrate_947523_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a908ce4-a357-4bd0-bdd6-5e1f76d02f58_calibrate_947523_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5e07970d-16e9-42db-8767-66a80de058c9_characterizeImage_947523_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5e07970d-16e9-42db-8767-66a80de058c9_characterizeImage_947523_8.stderr
parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4bb5c44042d667061108fed2e4949bd with result from task 517
parsl.dataflow.dflow DEBUG: Task 517 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 518 has memoization hash d25263a6050e3a1a5978d9ac648f12f5
parsl.dataflow.memoization INFO: Task 518 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 963
parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2aed7856-786b-446d-af11-5d80f7c6229e_calibrate_947523_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2aed7856-786b-446d-af11-5d80f7c6229e_calibrate_947523_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/231e88ef-4965-49ca-a398-d22918e4340d_characterizeImage_947523_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/231e88ef-4965-49ca-a398-d22918e4340d_characterizeImage_947523_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:49 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 71211f679387cf4f64cd0bb3e515a50f with result from task 586
parsl.dataflow.dflow DEBUG: Task 586 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 587 has memoization hash e4fb98d31c0935280ed9c2b1e3bbfe17
parsl.dataflow.memoization INFO: Task 587 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 964
parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/273e9131-ebfd-4787-ae0e-44fabb44fc47_calibrate_947523_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/273e9131-ebfd-4787-ae0e-44fabb44fc47_calibrate_947523_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7215405d-a113-460c-8bea-53ce36b7cba3_characterizeImage_947523_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/7215405d-a113-460c-8bea-53ce36b7cba3_characterizeImage_947523_1.stderr
parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd38cf2d63febb8243bd345f9cc1cdc9 with result from task 724
parsl.dataflow.dflow DEBUG: Task 724 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 725 has memoization hash bcd60ff1227261ddbed05d6616b48fb5
parsl.dataflow.memoization INFO: Task 725 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e75e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 965
parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5ddcb4fd-f85f-4be0-b672-aa4311cb0f2a_calibrate_947526_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5ddcb4fd-f85f-4be0-b672-aa4311cb0f2a_calibrate_947526_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c2622d53-2605-4459-abc3-a54f887bf020_characterizeImage_947526_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c2622d53-2605-4459-abc3-a54f887bf020_characterizeImage_947526_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:50 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b46d32649a7db629377d7b314d815e23 with result from task 799
parsl.dataflow.dflow DEBUG: Task 799 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 8478bc04e71a4520db5d87457e60cbc9
parsl.dataflow.memoization INFO: Task 800 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 966
parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dcae169a-0851-42b7-bd92-07147e6daa3b_calibrate_947524_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dcae169a-0851-42b7-bd92-07147e6daa3b_calibrate_947524_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cee2c7f7-580f-43df-a92d-02aba249eff6_characterizeImage_947524_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cee2c7f7-580f-43df-a92d-02aba249eff6_characterizeImage_947524_55.stderr
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb55a2addb1f49eff44c83900bcebb95 with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash ca0e91efdac085888ae83762cd202e4d
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 967
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f6785c32-23be-4aad-bff3-8b7ac5ee5c1b_calibrate_947523_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f6785c32-23be-4aad-bff3-8b7ac5ee5c1b_calibrate_947523_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6cf5b653-a70d-4f3b-a80b-c492fb17ca54_characterizeImage_947523_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6cf5b653-a70d-4f3b-a80b-c492fb17ca54_characterizeImage_947523_51.stderr
parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 414701d1aa3219f5ecf8009ed7e41fa4 with result from task 379
parsl.dataflow.dflow DEBUG: Task 379 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 380 has memoization hash cc91776784886ecbe5e2bc118a3edcda
parsl.dataflow.memoization INFO: Task 380 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 968
parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d88907cd-299b-4fc3-9231-8d8e09865b98_calibrate_947418_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d88907cd-299b-4fc3-9231-8d8e09865b98_calibrate_947418_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6682103b-48a3-433d-835f-559cd0235720_characterizeImage_947418_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6682103b-48a3-433d-835f-559cd0235720_characterizeImage_947418_13.stderr
parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ce99f5856d5a34471f5d10f016b4860 with result from task 703
parsl.dataflow.dflow DEBUG: Task 703 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 704 has memoization hash bd8b0df220b09d0a8eb846d0213f400e
parsl.dataflow.memoization INFO: Task 704 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 969
parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5a3325fa-dff2-42bb-9d8f-b66e80d1c459_calibrate_947420_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5a3325fa-dff2-42bb-9d8f-b66e80d1c459_calibrate_947420_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/6a4f5c56-7199-46da-9e8e-336b287fc5b9_characterizeImage_947420_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/6a4f5c56-7199-46da-9e8e-336b287fc5b9_characterizeImage_947420_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:50 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:51 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2f482c92e152d2cc289ad5adc21c730 with result from task 871
parsl.dataflow.dflow DEBUG: Task 871 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 872 has memoization hash b7452aa73e875d990a9088c5b6e5c2c1
parsl.dataflow.memoization INFO: Task 872 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 970
parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/79a2bc45-b4bf-4674-bf7d-b1b6ac5ae337_calibrate_947420_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/79a2bc45-b4bf-4674-bf7d-b1b6ac5ae337_calibrate_947420_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9afdde3c-37eb-48d2-9016-959261e95f19_characterizeImage_947420_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/9afdde3c-37eb-48d2-9016-959261e95f19_characterizeImage_947420_32.stderr
parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aedeca590610532ee0b0f6acf264f9e0 with result from task 832
parsl.dataflow.dflow DEBUG: Task 832 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 833 has memoization hash cae56f5c7337f80ea355e54d58e9165c
parsl.dataflow.memoization INFO: Task 833 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 971
parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/660d3e08-aa2b-4b18-b5bc-646432fd4e60_calibrate_947418_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/660d3e08-aa2b-4b18-b5bc-646432fd4e60_calibrate_947418_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/f6fcca11-11ff-485a-8d63-5e069bc52883_characterizeImage_947418_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/f6fcca11-11ff-485a-8d63-5e069bc52883_characterizeImage_947418_52.stderr
parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5d953832bf9abfe2c2c815eb7a9383d with result from task 1066
parsl.dataflow.dflow DEBUG: Task 1066 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 7c4715ef35231e8db6ca1022e14fe257
parsl.dataflow.memoization INFO: Task 1067 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 972
parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9a62b401-bddd-4290-b65f-542bdc9c5cc5_calibrate_947524_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9a62b401-bddd-4290-b65f-542bdc9c5cc5_calibrate_947524_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6b7446b5-8529-4fea-b610-f332e19e4195_characterizeImage_947524_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6b7446b5-8529-4fea-b610-f332e19e4195_characterizeImage_947524_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:51 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6809fbd0b4e460619b07fb9e75f2c215 with result from task 469
parsl.dataflow.dflow DEBUG: Task 469 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 470 has memoization hash a6ad410cd0db13250d17d9deaa71efe5
parsl.dataflow.memoization INFO: Task 470 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 973
parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/777795b7-4dda-4daf-8c88-cc36bfc23f84_calibrate_947420_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/777795b7-4dda-4daf-8c88-cc36bfc23f84_calibrate_947420_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a475c957-2a9c-42fc-bbe5-31c7b616f448_characterizeImage_947420_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/a475c957-2a9c-42fc-bbe5-31c7b616f448_characterizeImage_947420_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:51 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8bfd1bcd166990459cfc68944339b38 with result from task 955
parsl.dataflow.dflow DEBUG: Task 955 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 956 has memoization hash c4aa61aa71ec93e4e580809be7a38f5f
parsl.dataflow.memoization INFO: Task 956 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 974
parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8e5b8ab7-5f54-4bfc-89c6-c5b348caddc8_calibrate_947420_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8e5b8ab7-5f54-4bfc-89c6-c5b348caddc8_calibrate_947420_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/76a474de-ff67-462e-9cd8-75e0fdd19290_characterizeImage_947420_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/76a474de-ff67-462e-9cd8-75e0fdd19290_characterizeImage_947420_10.stderr
parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0984d7e60638aead998944ca96bf7a59 with result from task 739
parsl.dataflow.dflow DEBUG: Task 739 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 0949c9aef05ff716f0dee27f5763fa92
parsl.dataflow.memoization INFO: Task 740 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 975
parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b2ec7803-34dd-451e-8b89-16d3bc29702f_calibrate_947422_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b2ec7803-34dd-451e-8b89-16d3bc29702f_calibrate_947422_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/be692336-b29d-4f73-b386-822817191a3b_characterizeImage_947422_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/be692336-b29d-4f73-b386-822817191a3b_characterizeImage_947422_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:52 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da6bb99445a12341a0e1281811ed5d51 with result from task 835
parsl.dataflow.dflow DEBUG: Task 835 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 42de8e80a901f4edb75c8425b003fad8
parsl.dataflow.memoization INFO: Task 836 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 976
parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1c60deba-0570-4fde-88d1-1f78c4f8166d_calibrate_947418_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1c60deba-0570-4fde-88d1-1f78c4f8166d_calibrate_947418_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7895281d-8d15-4e30-bd08-a965ac148931_characterizeImage_947418_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7895281d-8d15-4e30-bd08-a965ac148931_characterizeImage_947418_57.stderr
parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d71b23fdfbc92b2f032d5be7e21ec70 with result from task 673
parsl.dataflow.dflow DEBUG: Task 673 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 027f5ba53fd7b19f8a17c57ff9880e63
parsl.dataflow.memoization INFO: Task 674 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 977
parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e374fdd1-928d-4a47-8c49-a0c0b86b63cf_calibrate_947422_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e374fdd1-928d-4a47-8c49-a0c0b86b63cf_calibrate_947422_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/12d5d1ba-8d7a-440a-9305-11b46d7aa5ce_characterizeImage_947422_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/12d5d1ba-8d7a-440a-9305-11b46d7aa5ce_characterizeImage_947422_29.stderr
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6220c97792e1dd75f6d10f06e9f4323c with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/85fb9679-740e-4b0c-bf2d-1c3e0343ba05_calibrate_947523_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/85fb9679-740e-4b0c-bf2d-1c3e0343ba05_calibrate_947523_57.stderr
parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 274fd0c6fb5bee99c13c9009379ec76e with result from task 265
parsl.dataflow.dflow DEBUG: Task 265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 7c2667fdcc48a6285f60aeb0a6e9c102
parsl.dataflow.memoization INFO: Task 266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 978
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/79a3bde7-8b71-4eef-bbc3-ff0399da738c_calibrate_947526_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/79a3bde7-8b71-4eef-bbc3-ff0399da738c_calibrate_947526_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/777be1ce-f60a-4288-99e8-718cf90a4db2_characterizeImage_947526_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/777be1ce-f60a-4288-99e8-718cf90a4db2_characterizeImage_947526_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:52 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec7722f3643a9303e4b77faba46de362 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash a9405ebac9d37ff1fabe96b07ca35432
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 979
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/2792f5a7-6c4d-4e4a-9bac-5cca371f9c65_calibrate_947526_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/2792f5a7-6c4d-4e4a-9bac-5cca371f9c65_calibrate_947526_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/21f9be78-c3fb-4e48-9e11-ef3de58a00ca_characterizeImage_947526_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/21f9be78-c3fb-4e48-9e11-ef3de58a00ca_characterizeImage_947526_13.stderr
parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e58dc1a4a8dbfb02c08eae97746c874 with result from task 700
parsl.dataflow.dflow DEBUG: Task 700 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 701 has memoization hash a6bedbd974709293ad328bc5fc5ae201
parsl.dataflow.memoization INFO: Task 701 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 980
parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5038707c-3058-4a1f-8cb7-d4d6cfbd2647_calibrate_947420_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5038707c-3058-4a1f-8cb7-d4d6cfbd2647_calibrate_947420_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/91409b30-97b6-4588-a1fe-88515c5b17ec_characterizeImage_947420_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/91409b30-97b6-4588-a1fe-88515c5b17ec_characterizeImage_947420_1.stderr
parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1660065923c7ec4e583bbde972be9de0 with result from task 502
parsl.dataflow.dflow DEBUG: Task 502 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 503 has memoization hash c14ba264c9baedbfdb7aa6388f36facf
parsl.dataflow.memoization INFO: Task 503 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 981
parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/611d1fb3-8a41-4b22-9b7d-c57c3b764fa5_calibrate_947528_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/611d1fb3-8a41-4b22-9b7d-c57c3b764fa5_calibrate_947528_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/d8bc5c8a-367d-470d-844d-d0d2c28ae4f8_characterizeImage_947528_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/d8bc5c8a-367d-470d-844d-d0d2c28ae4f8_characterizeImage_947528_1.stderr
parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26fbdaf691b05d66b42ee78021a29118 with result from task 655
parsl.dataflow.dflow DEBUG: Task 655 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 875086ce3fecc977d38147635d9a9bda
parsl.dataflow.memoization INFO: Task 656 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 982
parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/621581a6-6eed-4ab8-89dc-e9567f5134ec_calibrate_947523_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/621581a6-6eed-4ab8-89dc-e9567f5134ec_calibrate_947523_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ca1ed468-d589-4013-ae44-4ae7c85afa6a_characterizeImage_947523_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ca1ed468-d589-4013-ae44-4ae7c85afa6a_characterizeImage_947523_4.stderr
parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbe2e08ba5b2fd49b20a0e2bbef232ba with result from task 190
parsl.dataflow.dflow DEBUG: Task 190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 7f2da7ba8425650b55538223bec47cff
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 983
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f0f90091-3c67-40a0-86e3-b5db5067408b_calibrate_947523_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f0f90091-3c67-40a0-86e3-b5db5067408b_calibrate_947523_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1a11717-1059-4ca9-8f99-5661a92b2150_characterizeImage_947523_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1a11717-1059-4ca9-8f99-5661a92b2150_characterizeImage_947523_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:52 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79f8fcdd82a4d34c9918e2a16749dd1a with result from task 568
parsl.dataflow.dflow DEBUG: Task 568 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 569 has memoization hash e8f2c5ce10d60656ea42fca41130e95a
parsl.dataflow.memoization INFO: Task 569 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 984
parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9168b06b-b297-461b-b06d-627a748a9d27_calibrate_947420_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9168b06b-b297-461b-b06d-627a748a9d27_calibrate_947420_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/60239d29-7502-4917-b05b-3bea7460301c_characterizeImage_947420_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/60239d29-7502-4917-b05b-3bea7460301c_characterizeImage_947420_12.stderr
parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c7f71dd9a04f20e9a636c92cc6f138e with result from task 412
parsl.dataflow.dflow DEBUG: Task 412 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 413 has memoization hash e529b150ba626372a73492955867ec8f
parsl.dataflow.memoization INFO: Task 413 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 985
parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1c76ea0c-6a27-49da-85b7-4ce9c52985c8_calibrate_947523_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1c76ea0c-6a27-49da-85b7-4ce9c52985c8_calibrate_947523_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a3152ebd-8eac-498e-b1e3-84df30e6460e_characterizeImage_947523_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a3152ebd-8eac-498e-b1e3-84df30e6460e_characterizeImage_947523_20.stderr
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcff50dc1e73932270c5eef16c9e1a73 with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 33f44775b4e73cf9619c1ffa6ac7ff68
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 986
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/25f7e71f-48e1-4b9c-bb9f-f1c36202cd1d_calibrate_947528_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/25f7e71f-48e1-4b9c-bb9f-f1c36202cd1d_calibrate_947528_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5a32c7a6-1408-4fe6-b546-657eae51ef23_characterizeImage_947528_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5a32c7a6-1408-4fe6-b546-657eae51ef23_characterizeImage_947528_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 425
parsl.dataflow.strategy DEBUG: Executor multi has 425 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 425
parsl.dataflow.strategy DEBUG: Executor multi has 425 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 117474f87c599340f246d2658905a74d with result from task 682
parsl.dataflow.dflow DEBUG: Task 682 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 683 has memoization hash c946e80274c9e3c49f9ed11c976e7e39
parsl.dataflow.memoization INFO: Task 683 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 987
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b994f19f-9243-4037-b8e6-8b682156feee_calibrate_947528_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b994f19f-9243-4037-b8e6-8b682156feee_calibrate_947528_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fb08e93c-c322-4999-bb5c-e1ba2e0fcd22_characterizeImage_947528_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fb08e93c-c322-4999-bb5c-e1ba2e0fcd22_characterizeImage_947528_62.stderr
parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d39751b343f99d9cc9df3b9bb8d52a3a with result from task 196
parsl.dataflow.dflow DEBUG: Task 196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 197 has memoization hash d3ce8f2716401a048a0d82ee72e56267
parsl.dataflow.memoization INFO: Task 197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 988
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9293aa8a-2656-4501-a2a9-bed79196edec_calibrate_947420_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9293aa8a-2656-4501-a2a9-bed79196edec_calibrate_947420_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e3260a34-25cb-49f2-9a2d-8e194df0fec4_characterizeImage_947420_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e3260a34-25cb-49f2-9a2d-8e194df0fec4_characterizeImage_947420_50.stderr
parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1035dec94ecf856bbdf73fa63ccfeee with result from task 1051
parsl.dataflow.dflow DEBUG: Task 1051 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash e8778565c95cfe07128d07ee26808fe4
parsl.dataflow.memoization INFO: Task 1052 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 989
parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/7e8629a8-6bd6-492f-a7ef-33458866faa9_calibrate_947528_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/7e8629a8-6bd6-492f-a7ef-33458866faa9_calibrate_947528_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/63c30d0c-a705-4e69-ba3c-215ff2d982ef_characterizeImage_947528_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/63c30d0c-a705-4e69-ba3c-215ff2d982ef_characterizeImage_947528_49.stderr
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0bfe0ddf18056d0ee0ffa0e92256462c with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash a7256ed985466c2b86bb6c8269bb7784
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e968e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 990
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dfec4333-7203-4eb6-b3fd-4b4461dc2a77_calibrate_947418_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dfec4333-7203-4eb6-b3fd-4b4461dc2a77_calibrate_947418_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d0852d6a-42ce-4049-8c88-995fe0f7ee1c_characterizeImage_947418_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d0852d6a-42ce-4049-8c88-995fe0f7ee1c_characterizeImage_947418_12.stderr
parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5e6885770f5cd9f0e9f4f02907a9e36 with result from task 1054
parsl.dataflow.dflow DEBUG: Task 1054 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 034985de074405d16f0c35e380bff0b7
parsl.dataflow.memoization INFO: Task 1055 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 991
parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f3156a5e-a4e4-4413-825a-2382eb13d7f3_calibrate_947523_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f3156a5e-a4e4-4413-825a-2382eb13d7f3_calibrate_947523_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3390dacc-fdb8-4d01-8827-f10a14488192_characterizeImage_947523_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3390dacc-fdb8-4d01-8827-f10a14488192_characterizeImage_947523_7.stderr
parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23ce941d6bf4f4c6aedf5fcecfb91f1b with result from task 748
parsl.dataflow.dflow DEBUG: Task 748 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 51efdc48f176285545c36605f12cc5d3
parsl.dataflow.memoization INFO: Task 749 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 992
parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5e14787e-45bc-440c-aa74-e66129e82c9c_calibrate_947523_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5e14787e-45bc-440c-aa74-e66129e82c9c_calibrate_947523_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/e08b4bad-7b4c-4d66-a053-e02972feabfe_characterizeImage_947523_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/e08b4bad-7b4c-4d66-a053-e02972feabfe_characterizeImage_947523_38.stderr
parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8f634dea45932bdf5a08ac628236ea9 with result from task 262
parsl.dataflow.dflow DEBUG: Task 262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 263 has memoization hash abc19fbfb6709871967685de9bdaa34c
parsl.dataflow.memoization INFO: Task 263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 993
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e014fcb1-a3c4-4cb3-8152-d458be2dceac_calibrate_947526_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e014fcb1-a3c4-4cb3-8152-d458be2dceac_calibrate_947526_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/6590c257-5d37-440c-9f48-14cf6784d949_characterizeImage_947526_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/6590c257-5d37-440c-9f48-14cf6784d949_characterizeImage_947526_57.stderr
parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2801331b19ee2857a0f5302a4b9d9973 with result from task 187
parsl.dataflow.dflow DEBUG: Task 187 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 0e1fc90671eaa0a7864bd7f408fb5249
parsl.dataflow.memoization INFO: Task 188 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 994
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ad3ca06d-6099-4dbd-93cc-962cfb8ff8d1_calibrate_947420_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ad3ca06d-6099-4dbd-93cc-962cfb8ff8d1_calibrate_947420_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/464ab7bd-71d0-4f64-8084-70cdc13f2f69_characterizeImage_947420_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/464ab7bd-71d0-4f64-8084-70cdc13f2f69_characterizeImage_947420_4.stderr
parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9e1c6627a601be48a157fd57f4bb5fa with result from task 355
parsl.dataflow.dflow DEBUG: Task 355 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 356 has memoization hash e0cb44c520d7eef493581d34d6427a36
parsl.dataflow.memoization INFO: Task 356 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 995
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/87ce2a96-bdb4-4702-979b-b0e1f1601548_calibrate_947526_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/87ce2a96-bdb4-4702-979b-b0e1f1601548_calibrate_947526_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/97a08412-8d7a-44f2-b603-e328921b758e_characterizeImage_947526_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/97a08412-8d7a-44f2-b603-e328921b758e_characterizeImage_947526_54.stderr
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03179fa9c2a6477dc76c111cb6de8379 with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 5ae4bc0b92428c708785e507ef92801d
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 996
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/e18999b7-cb9d-4126-a82f-2f2fcd384a40_calibrate_947420_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/e18999b7-cb9d-4126-a82f-2f2fcd384a40_calibrate_947420_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/43c6985a-cf38-452d-a2b2-08d4646e4acd_characterizeImage_947420_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/43c6985a-cf38-452d-a2b2-08d4646e4acd_characterizeImage_947420_8.stderr
parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc424e1dd99f251a18d0898e8aff1761 with result from task 628
parsl.dataflow.dflow DEBUG: Task 628 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 629 has memoization hash d4fb68848712440a4d0f2c2bffe78267
parsl.dataflow.memoization INFO: Task 629 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 997
parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/75b2b74a-6439-4da4-bf21-ca4f7431dc19_calibrate_947422_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/75b2b74a-6439-4da4-bf21-ca4f7431dc19_calibrate_947422_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/48685365-85fd-497b-85f9-dd7516aa6fe0_characterizeImage_947422_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/48685365-85fd-497b-85f9-dd7516aa6fe0_characterizeImage_947422_49.stderr
parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f836fe6ba9a0f68d4402433f77caa4eb with result from task 284
parsl.dataflow.dflow DEBUG: Task 284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c5122035-ea9e-4f8f-8002-3f44630cebd2_calibrate_947418_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c5122035-ea9e-4f8f-8002-3f44630cebd2_calibrate_947418_51.stderr
parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62eb9c59a9b4d531de9decd9d8ef1a1d with result from task 217
parsl.dataflow.dflow DEBUG: Task 217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 28f7e6c180d35989cbad943b1aec7282
parsl.dataflow.memoization INFO: Task 218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 998
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/aa86da19-c3d6-472f-952b-93e83efee985_calibrate_947418_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/aa86da19-c3d6-472f-952b-93e83efee985_calibrate_947418_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/08d26375-1d48-435f-b393-4f3075e27ea3_characterizeImage_947418_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/08d26375-1d48-435f-b393-4f3075e27ea3_characterizeImage_947418_47.stderr
parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9d86ead6ad9dd43a37e8f81dcfc9f83 with result from task 439
parsl.dataflow.dflow DEBUG: Task 439 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 5371f8f97f1825c1c1e3e26383004092
parsl.dataflow.memoization INFO: Task 440 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 999
parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/8230e221-310f-4528-b4ae-da9774fae092_calibrate_947526_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/8230e221-310f-4528-b4ae-da9774fae092_calibrate_947526_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/72d3b1bd-bf14-4e55-8942-0a0b855788de_characterizeImage_947526_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/72d3b1bd-bf14-4e55-8942-0a0b855788de_characterizeImage_947526_3.stderr
parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b4835a49b4250348d9a1d17eaf5e248 with result from task 397
parsl.dataflow.dflow DEBUG: Task 397 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 97ad4d5bd94ef5da3edc7a934a8b4e54
parsl.dataflow.memoization INFO: Task 398 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 1000
parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/669a1001-a4d6-476d-9df1-c87d76105e83_calibrate_947420_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/669a1001-a4d6-476d-9df1-c87d76105e83_calibrate_947420_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/53dc7927-573a-43af-8827-bf1ad43d23fd_characterizeImage_947420_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/53dc7927-573a-43af-8827-bf1ad43d23fd_characterizeImage_947420_34.stderr
parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3d8b4dddd6df9a4647663cb7a74f7d with result from task 391
parsl.dataflow.dflow DEBUG: Task 391 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 6eeea93e7497f05092ee951ddaaa5234
parsl.dataflow.memoization INFO: Task 392 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 1001
parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/08ca40b8-e876-406f-8af6-10dd5ce2f1f6_calibrate_947420_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/08ca40b8-e876-406f-8af6-10dd5ce2f1f6_calibrate_947420_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/65d669c7-3719-48ac-a2be-20bf562a9f7a_characterizeImage_947420_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/65d669c7-3719-48ac-a2be-20bf562a9f7a_characterizeImage_947420_22.stderr
parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 711ce2444e91d9ad8b1ef698e82559c0 with result from task 1000
parsl.dataflow.dflow DEBUG: Task 1000 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 51de3fcdefc7dedf796247f9f74d759b
parsl.dataflow.memoization INFO: Task 1001 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1002
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/01f2c3c7-b53a-4e69-b92e-f62c474de019_calibrate_947422_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/01f2c3c7-b53a-4e69-b92e-f62c474de019_calibrate_947422_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ce772f1f-5c69-4401-be44-50b6d15ec488_characterizeImage_947422_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ce772f1f-5c69-4401-be44-50b6d15ec488_characterizeImage_947422_15.stderr
parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97d9cee766808b8e006ab023c748e720 with result from task 712
parsl.dataflow.dflow DEBUG: Task 712 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 713 has memoization hash b5d86e816ad926b7cc6d7eee61b893ee
parsl.dataflow.memoization INFO: Task 713 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 1003
parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e40fffed-db4c-4282-805a-cdfc335e97dc_calibrate_947523_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e40fffed-db4c-4282-805a-cdfc335e97dc_calibrate_947523_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1e2fd78-0200-47ad-b600-b389e6732d81_characterizeImage_947523_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/b1e2fd78-0200-47ad-b600-b389e6732d81_characterizeImage_947523_44.stderr
parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55a25a65ab935ea2138243d7a2be8fad with result from task 772
parsl.dataflow.dflow DEBUG: Task 772 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 689a929ba92da761d7d851103fafa457
parsl.dataflow.memoization INFO: Task 773 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 1004
parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/835b39ff-5ab5-48c0-8e2e-d6b8b64a8d32_calibrate_947420_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/835b39ff-5ab5-48c0-8e2e-d6b8b64a8d32_calibrate_947420_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be049c80-c8f5-46fd-abbe-ea0f01320356_characterizeImage_947420_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/be049c80-c8f5-46fd-abbe-ea0f01320356_characterizeImage_947420_39.stderr
parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91bb7a2fe45a58ea8573b0840c81e077 with result from task 665
parsl.dataflow.dflow DEBUG: Task 665 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e411f766-31e9-42c3-a130-78d2f855afa0_calibrate_947422_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e411f766-31e9-42c3-a130-78d2f855afa0_calibrate_947422_31.stderr
parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b2556275f85c8fa7704e4fd2ef1a8ba with result from task 817
parsl.dataflow.dflow DEBUG: Task 817 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 337e0ee42044b841f6ce72d658da9544
parsl.dataflow.memoization INFO: Task 818 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 1005
parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/58ef388a-48fc-44a8-91be-ef08a2bbf375_calibrate_947524_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/58ef388a-48fc-44a8-91be-ef08a2bbf375_calibrate_947524_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7ab48534-8386-4da4-ac70-894611b3f81c_characterizeImage_947524_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7ab48534-8386-4da4-ac70-894611b3f81c_characterizeImage_947524_48.stderr
parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f47a53f776e488aedb732f793abe3405 with result from task 637
parsl.dataflow.dflow DEBUG: Task 637 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 60cb6892fee933664043040d90f45ce4
parsl.dataflow.memoization INFO: Task 638 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 1006
parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f20df706-294f-4b1b-b2a8-f8d6c86ce07e_calibrate_947420_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f20df706-294f-4b1b-b2a8-f8d6c86ce07e_calibrate_947420_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e5ca0c2f-9fff-4ebd-b3db-0a6a3a4d01e0_characterizeImage_947420_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e5ca0c2f-9fff-4ebd-b3db-0a6a3a4d01e0_characterizeImage_947420_41.stderr
parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55405181dca56b6301ee48d6fd47ad4c with result from task 1070
parsl.dataflow.dflow DEBUG: Task 1070 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/378ad62e-b585-427a-b4b7-1362db032b8f_calibrate_947420_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/378ad62e-b585-427a-b4b7-1362db032b8f_calibrate_947420_49.stderr
parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 03c2e6ec5be85eccd11351c9da58da78 with result from task 646
parsl.dataflow.dflow DEBUG: Task 646 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 79c6e21ee28ea191501d6f67bbc021d1
parsl.dataflow.memoization INFO: Task 647 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 1007
parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a372aec-cd86-42a6-b590-e5d811b3cd7c_calibrate_947422_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a372aec-cd86-42a6-b590-e5d811b3cd7c_calibrate_947422_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2287180e-0227-4a47-8eca-b88583de519e_characterizeImage_947422_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/2287180e-0227-4a47-8eca-b88583de519e_characterizeImage_947422_56.stderr
parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e41b9bc5eba37e19bd1c057e916c6b0 with result from task 346
parsl.dataflow.dflow DEBUG: Task 346 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 347 has memoization hash aa6e1713d4e55610b628c076b83a0f91
parsl.dataflow.memoization INFO: Task 347 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f21120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 1008
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8d5ddaaa-5842-4ed6-adf2-9b310588c043_calibrate_947524_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8d5ddaaa-5842-4ed6-adf2-9b310588c043_calibrate_947524_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d177ebed-29ad-42d1-a3d6-d2fe0d4ab53b_characterizeImage_947524_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d177ebed-29ad-42d1-a3d6-d2fe0d4ab53b_characterizeImage_947524_33.stderr
parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15100f163a195fffadc096dae8eb75bd with result from task 679
parsl.dataflow.dflow DEBUG: Task 679 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 680 has memoization hash ca2fba3ff01344122fe2feb4ada20e4f
parsl.dataflow.memoization INFO: Task 680 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e36f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 1009
parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/ff3d3c31-f6f9-49d5-b046-c27e66917e2c_calibrate_947422_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/ff3d3c31-f6f9-49d5-b046-c27e66917e2c_calibrate_947422_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f4c7c2f5-5ae3-40b1-946d-889e2a8dc698_characterizeImage_947422_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f4c7c2f5-5ae3-40b1-946d-889e2a8dc698_characterizeImage_947422_55.stderr
parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 17d1fd476a55139c5b44860463d7a9d7 with result from task 677
parsl.dataflow.dflow DEBUG: Task 677 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ed589bc3-bda7-4745-a3c9-f3739d8e42c7_calibrate_947418_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ed589bc3-bda7-4745-a3c9-f3739d8e42c7_calibrate_947418_55.stderr
parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d49d00ca7244ef1094a60834f54f211 with result from task 514
parsl.dataflow.dflow DEBUG: Task 514 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 515 has memoization hash f7ca281cfff6cc901b900bd5e3500d92
parsl.dataflow.memoization INFO: Task 515 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 1010
parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5360db-cf1a-4e0e-b4a5-cc6c48ba5c2b_calibrate_947526_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5360db-cf1a-4e0e-b4a5-cc6c48ba5c2b_calibrate_947526_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/db93699a-7a00-486e-af52-10cfa04e899b_characterizeImage_947526_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/db93699a-7a00-486e-af52-10cfa04e899b_characterizeImage_947526_49.stderr
parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a459f6cad363c247ef98a2f466ffc0d1 with result from task 1099
parsl.dataflow.dflow DEBUG: Task 1099 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash c5e40e3dbd0bcac3f72d6c9e5b30277f
parsl.dataflow.memoization INFO: Task 1100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 1011
parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/07eaadda-3d87-4a5b-96df-93735c2ab836_calibrate_947523_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/07eaadda-3d87-4a5b-96df-93735c2ab836_calibrate_947523_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/25a15a81-31dd-4d29-a258-6f087de5e17f_characterizeImage_947523_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/25a15a81-31dd-4d29-a258-6f087de5e17f_characterizeImage_947523_37.stderr
parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 307e3c3e9771205b4083494c98a058ea with result from task 205
parsl.dataflow.dflow DEBUG: Task 205 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 39692de7e0c8dd1e29e593e5cf0fc604
parsl.dataflow.memoization INFO: Task 206 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 1012
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/dac0b60a-43bb-4aa8-aa60-34415d2730de_calibrate_947523_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/dac0b60a-43bb-4aa8-aa60-34415d2730de_calibrate_947523_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/f267abe3-1593-4db0-b71d-7abe9a7c4bbc_characterizeImage_947523_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/f267abe3-1593-4db0-b71d-7abe9a7c4bbc_characterizeImage_947523_48.stderr
2024-03-14 07:27:58 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:27:58 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9dc61d4b89caea9a42648f4a04cf3429 with result from task 862
parsl.dataflow.dflow DEBUG: Task 862 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 863 has memoization hash a6762e1fc5e44fb068350b5ac5bb0f93
parsl.dataflow.memoization INFO: Task 863 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 1013
parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ce5772ca-3b50-4562-b218-3f4d7702c5c6_calibrate_947420_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ce5772ca-3b50-4562-b218-3f4d7702c5c6_calibrate_947420_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af64cb5f-7c4e-42fa-8edb-bb6d090bfe73_characterizeImage_947420_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af64cb5f-7c4e-42fa-8edb-bb6d090bfe73_characterizeImage_947420_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:58 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4d197240efd7cb7828362fb7ae5422f with result from task 592
parsl.dataflow.dflow DEBUG: Task 592 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 68031a07ba26ea1cc9af78b699f19292
parsl.dataflow.memoization INFO: Task 593 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 1014
parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/896815f6-0b8c-4ec5-aecd-7ec6615dc61e_calibrate_947422_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/896815f6-0b8c-4ec5-aecd-7ec6615dc61e_calibrate_947422_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/91bcd2a8-7cbf-4298-bea5-4404c1bd5074_characterizeImage_947422_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/91bcd2a8-7cbf-4298-bea5-4404c1bd5074_characterizeImage_947422_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:58 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ffae205ec443f89cda6ea0db765051f with result from task 320
parsl.dataflow.dflow DEBUG: Task 320 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2b284254-e29c-48db-ab88-b9898627de91_calibrate_947420_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2b284254-e29c-48db-ab88-b9898627de91_calibrate_947420_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:58 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:58 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52201ba983121e6344066f61a8d31e6a with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 8d4597cba808246903cf20c3f849bc55
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 1015
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/811f1078-a0e3-459a-8fd6-a520e4d76539_calibrate_947420_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/811f1078-a0e3-459a-8fd6-a520e4d76539_calibrate_947420_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/8fd17842-7b45-45de-97e7-99441a69c70c_characterizeImage_947420_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/8fd17842-7b45-45de-97e7-99441a69c70c_characterizeImage_947420_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:59 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d7a5691512dde946d0867937ef649ea with result from task 1087
parsl.dataflow.dflow DEBUG: Task 1087 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash 9a1a62d768e8fa878089990070d2db04
parsl.dataflow.memoization INFO: Task 1088 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1016
parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/59aedb6d-eac8-4e69-8779-0c445e5ea7e6_calibrate_947528_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/59aedb6d-eac8-4e69-8779-0c445e5ea7e6_calibrate_947528_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/049b854d-7cc1-4c4b-b6fe-c8c5b4335793_characterizeImage_947528_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/049b854d-7cc1-4c4b-b6fe-c8c5b4335793_characterizeImage_947528_9.stderr
parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9d83cc0e374bdeb6605e77e80d63f8 with result from task 472
parsl.dataflow.dflow DEBUG: Task 472 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 473 has memoization hash bec4339be055e5b78e260e99c38e1f16
parsl.dataflow.memoization INFO: Task 473 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 1017
parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1ae12f04-8718-42b6-b315-44d3e4c689b9_calibrate_947524_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1ae12f04-8718-42b6-b315-44d3e4c689b9_calibrate_947524_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/138da8fc-69c9-4539-b110-5c5247326e98_characterizeImage_947524_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/138da8fc-69c9-4539-b110-5c5247326e98_characterizeImage_947524_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:59 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:59 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20b61fb74507be75703cd1c00572fedb with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash c2be245fb249d16b0e7a1ec6e2d266ce
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 1018
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/300f8f1f-e5d7-4f6f-80e5-533a29e11fc7_calibrate_947528_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/300f8f1f-e5d7-4f6f-80e5-533a29e11fc7_calibrate_947528_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7ffe30d6-f557-4a05-9f64-ecdca4cf38e7_characterizeImage_947528_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7ffe30d6-f557-4a05-9f64-ecdca4cf38e7_characterizeImage_947528_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:27:59 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f0f09d42ac3963abdeb62ecc678af65 with result from task 973
parsl.dataflow.dflow DEBUG: Task 973 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 7b41ddc712cd2fc7496bad08f42de0c7
parsl.dataflow.memoization INFO: Task 974 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 1019
parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/14cdcb45-a5b3-40e9-a024-1c8c350903ac_calibrate_947523_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/14cdcb45-a5b3-40e9-a024-1c8c350903ac_calibrate_947523_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6dd871e-45f8-4b11-bdf1-2381439d6caa_characterizeImage_947523_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/d6dd871e-45f8-4b11-bdf1-2381439d6caa_characterizeImage_947523_28.stderr
parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 75f371bfb84ab0898bdb6da9f06c67d8 with result from task 1063
parsl.dataflow.dflow DEBUG: Task 1063 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 207dd3e742e9107406a8a97781e89faa
parsl.dataflow.memoization INFO: Task 1064 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 1020
parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f348d33-bc9b-432b-8ebb-dd7d1d1dc375_calibrate_947528_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f348d33-bc9b-432b-8ebb-dd7d1d1dc375_calibrate_947528_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/dc371f14-2c74-437c-aa36-eb6badd2a9f3_characterizeImage_947528_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/dc371f14-2c74-437c-aa36-eb6badd2a9f3_characterizeImage_947528_3.stderr
parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dadaa3c5ea687321a57de92afd52a274 with result from task 868
parsl.dataflow.dflow DEBUG: Task 868 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 869 has memoization hash a2a9e80baaa7361d142614015cbc1b41
parsl.dataflow.memoization INFO: Task 869 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 1021
parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/fb8fe7c5-87f3-43b5-8832-13f4a63e8a37_calibrate_947526_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/fb8fe7c5-87f3-43b5-8832-13f4a63e8a37_calibrate_947526_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b67fd7f-485b-4384-884d-e4fb1fd4e39c_characterizeImage_947526_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b67fd7f-485b-4384-884d-e4fb1fd4e39c_characterizeImage_947526_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:00 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e30f35b52a85d9503dab935f373ec4ca with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash d5d303141e6f6c3f6402b0b2f50ad21f
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 1022
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/917fc988-75b6-4437-9356-b2d8d56fbfb2_calibrate_947418_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/917fc988-75b6-4437-9356-b2d8d56fbfb2_calibrate_947418_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1bb11128-8dad-4c29-a3af-0ff597cd523c_characterizeImage_947418_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1bb11128-8dad-4c29-a3af-0ff597cd523c_characterizeImage_947418_44.stderr
parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b8d58def7b739e980ea6850425b3b26 with result from task 769
parsl.dataflow.dflow DEBUG: Task 769 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 8e9c9cf946c9e9139fb49157735f8396
parsl.dataflow.memoization INFO: Task 770 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 1023
parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/0fd29e59-5ecb-4410-9cae-9eb59d0ba690_calibrate_947524_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/0fd29e59-5ecb-4410-9cae-9eb59d0ba690_calibrate_947524_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8c7c9be0-1485-48b5-a91c-5c6b8c19570a_characterizeImage_947524_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8c7c9be0-1485-48b5-a91c-5c6b8c19570a_characterizeImage_947524_42.stderr
parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c04b51adb4b98cc04c2cf1f09a677b6 with result from task 550
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:00 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow DEBUG: Task 550 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 4517300e08685bf303be2df17ac998e6
parsl.dataflow.memoization INFO: Task 551 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 1024
parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d08722c8-2055-4405-848b-1c1c9749df32_calibrate_947418_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d08722c8-2055-4405-848b-1c1c9749df32_calibrate_947418_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d52a49d3-b7d0-4686-8e45-adc19f6fde9e_characterizeImage_947418_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d52a49d3-b7d0-4686-8e45-adc19f6fde9e_characterizeImage_947418_56.stderr
parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f1312f17fe20f7311240e2b3aa00fd3f with result from task 1009
parsl.dataflow.dflow DEBUG: Task 1009 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash e818bd3e4e6635fa513a6a33e41ede0b
parsl.dataflow.memoization INFO: Task 1010 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1025
parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/de3ae67b-2f68-4b54-bdee-2daff2263af8_calibrate_947420_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/de3ae67b-2f68-4b54-bdee-2daff2263af8_calibrate_947420_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b0b7a480-6189-4c78-8e72-dc8a1b45b012_characterizeImage_947420_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/b0b7a480-6189-4c78-8e72-dc8a1b45b012_characterizeImage_947420_6.stderr
parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry abaa6c75c326fc757e3f6254f350704a with result from task 430
parsl.dataflow.dflow DEBUG: Task 430 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 431 has memoization hash c33888ba954eb638488ac2ca3b419f96
parsl.dataflow.memoization INFO: Task 431 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f528e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 1026
parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b25dde45-0a95-489a-8c63-f5b09e94973e_calibrate_947422_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b25dde45-0a95-489a-8c63-f5b09e94973e_calibrate_947422_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/43041829-f398-49cb-a705-ed43cb8aa5f0_characterizeImage_947422_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/43041829-f398-49cb-a705-ed43cb8aa5f0_characterizeImage_947422_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:00 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3de045d8cb19e70877801bd3a90d3e23 with result from task 844
parsl.dataflow.dflow DEBUG: Task 844 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 855c4ec8333e109da7bdb61ca28a8bf1
parsl.dataflow.memoization INFO: Task 845 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 1027
parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/377ceece-851a-4faf-8172-99103d7fb040_calibrate_947523_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/377ceece-851a-4faf-8172-99103d7fb040_calibrate_947523_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/702851f0-5475-467f-b99f-e156ccc829ec_characterizeImage_947523_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/702851f0-5475-467f-b99f-e156ccc829ec_characterizeImage_947523_5.stderr
parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e62ec62b9ec0e9bdedba2130bcd57284 with result from task 757
parsl.dataflow.dflow DEBUG: Task 757 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 758 has memoization hash e467dbefba4e5109ed380cba4a4f0112
parsl.dataflow.memoization INFO: Task 758 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 1028
parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1f83c109-6bec-47ca-9443-c5e1fff7c859_calibrate_947523_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1f83c109-6bec-47ca-9443-c5e1fff7c859_calibrate_947523_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/54db3523-7e7e-40d8-95cf-b805cf8e235f_characterizeImage_947523_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/54db3523-7e7e-40d8-95cf-b805cf8e235f_characterizeImage_947523_40.stderr
parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db05ccb68de03f8c23419eb93c129d9a with result from task 482
parsl.dataflow.dflow DEBUG: Task 482 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0586ddad-0baa-4ea7-972b-672e264dd6cf_calibrate_947418_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0586ddad-0baa-4ea7-972b-672e264dd6cf_calibrate_947418_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:01 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9665f8c9904b41ed1a73967870f4870 with result from task 433
parsl.dataflow.dflow DEBUG: Task 433 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 7c2104c79c76e4cc7f6073865b10d6e2
parsl.dataflow.memoization INFO: Task 434 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f52c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 1029
parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bacd90ef-3417-427e-95e7-0b2875fc2259_calibrate_947420_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bacd90ef-3417-427e-95e7-0b2875fc2259_calibrate_947420_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ea78427a-da0d-465d-b267-4486e6194623_characterizeImage_947420_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/ea78427a-da0d-465d-b267-4486e6194623_characterizeImage_947420_45.stderr
parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8179b7405c34125891758994c2388d with result from task 361
parsl.dataflow.dflow DEBUG: Task 361 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 362 has memoization hash b7ac30ed7457747c554b4203be802a22
parsl.dataflow.memoization INFO: Task 362 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 1030
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f96f2f8b-db2d-4851-bdea-7838156fed1e_calibrate_947420_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f96f2f8b-db2d-4851-bdea-7838156fed1e_calibrate_947420_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2860d712-3a21-463f-8936-58523f08b9bc_characterizeImage_947420_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/2860d712-3a21-463f-8936-58523f08b9bc_characterizeImage_947420_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:01 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38fa5ac9a2842e47d25dd0893eb3260e with result from task 323
parsl.dataflow.dflow DEBUG: Task 323 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e64345f-43dd-40fd-b85a-9aca5ddaf937_calibrate_947420_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e64345f-43dd-40fd-b85a-9aca5ddaf937_calibrate_947420_51.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5ee63d883835cdc888c3788fbfb0a8c with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash df2c58d352e419c115a0b14779890445
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 1031
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1bc934ca-0c07-4c6a-9411-93dcf3992a5b_calibrate_947524_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1bc934ca-0c07-4c6a-9411-93dcf3992a5b_calibrate_947524_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1a321fd6-822d-4596-af2c-ec61d1c44cae_characterizeImage_947524_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1a321fd6-822d-4596-af2c-ec61d1c44cae_characterizeImage_947524_44.stderr
parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 233a8ec01394f2967dd80e1d801977a5 with result from task 970
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:02 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow DEBUG: Task 970 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 6b6e059e272b7b1abd44fb1bc77d7de2
parsl.dataflow.memoization INFO: Task 971 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 1032
parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a1c3e144-f190-4659-a765-642736fd2768_calibrate_947422_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a1c3e144-f190-4659-a765-642736fd2768_calibrate_947422_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d1ad0f62-8fea-4e81-aa7a-ed5805afa87d_characterizeImage_947422_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d1ad0f62-8fea-4e81-aa7a-ed5805afa87d_characterizeImage_947422_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:02 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a37380b2574cff72fb7b437a24af852e with result from task 418
parsl.dataflow.dflow DEBUG: Task 418 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 419 has memoization hash a01388059596a972d0cdd243f97fd417
parsl.dataflow.memoization INFO: Task 419 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 1033
parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b56f5108-90db-4d45-87d7-eb6200484bda_calibrate_947528_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b56f5108-90db-4d45-87d7-eb6200484bda_calibrate_947528_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/313bd249-64ae-4f36-8fb3-6073fcf1663d_characterizeImage_947528_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/313bd249-64ae-4f36-8fb3-6073fcf1663d_characterizeImage_947528_32.stderr
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47e3502e0d47b70cc160c0a9f368419b with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 8f040c0ec54fe1e149fd1821305a74b0
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 1034
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5dd3f740-6b44-4ade-b5e0-c75a41f58968_calibrate_947526_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5dd3f740-6b44-4ade-b5e0-c75a41f58968_calibrate_947526_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b06a3720-a39e-429b-940f-c6c8bc60ccdd_characterizeImage_947526_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b06a3720-a39e-429b-940f-c6c8bc60ccdd_characterizeImage_947526_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:02 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:03 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24f19b5ebf6e25206f8671f9e2f8a8a1 with result from task 305
parsl.dataflow.dflow DEBUG: Task 305 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c8b5fd74-be4f-4420-a054-add026bd90dc_calibrate_947523_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c8b5fd74-be4f-4420-a054-add026bd90dc_calibrate_947523_31.stderr
parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1680e68c53582dd573b0ce1ab35fbc with result from task 277
parsl.dataflow.dflow DEBUG: Task 277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash d4f442cb27db804025006ef9375f35ad
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 1035
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0bdf7a2d-8589-42ff-bdb4-e793d33db83a_calibrate_947523_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0bdf7a2d-8589-42ff-bdb4-e793d33db83a_calibrate_947523_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3ed52454-78ed-4675-975e-05c6cf3c362f_characterizeImage_947523_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3ed52454-78ed-4675-975e-05c6cf3c362f_characterizeImage_947523_47.stderr
parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57cf70d01296b359f0c1c75227844f49 with result from task 337
parsl.dataflow.dflow DEBUG: Task 337 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 8590eb7b1dd6348e8bc5c5ce6c40c8da
parsl.dataflow.memoization INFO: Task 338 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 1036
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dbb0c4e7-7d1c-42f7-8d00-5b93a7e6454f_calibrate_947418_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dbb0c4e7-7d1c-42f7-8d00-5b93a7e6454f_calibrate_947418_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ecf6aed4-bc89-4593-b881-abd94961dc18_characterizeImage_947418_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ecf6aed4-bc89-4593-b881-abd94961dc18_characterizeImage_947418_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:03 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0f19da7b54a25e0d4fb3b31733e88c5 with result from task 719
parsl.dataflow.dflow DEBUG: Task 719 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/958334ad-80fc-48a2-b930-2bf643272bb1_calibrate_947420_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/958334ad-80fc-48a2-b930-2bf643272bb1_calibrate_947420_15.stderr
parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8374bd19cb934ac43d925b00dd4c921f with result from task 856
parsl.dataflow.dflow DEBUG: Task 856 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 857 has memoization hash ac40cd62bbc0de13de921e0536f96b83
parsl.dataflow.memoization INFO: Task 857 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 1037
parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/4d665bf3-686b-4e20-8f5e-9c28f0818228_calibrate_947523_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/4d665bf3-686b-4e20-8f5e-9c28f0818228_calibrate_947523_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8b409464-20b2-4eb8-a38c-fe6d7136f59b_characterizeImage_947523_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8b409464-20b2-4eb8-a38c-fe6d7136f59b_characterizeImage_947523_53.stderr
parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7ec5fde666151ac18315aef04148327 with result from task 808
parsl.dataflow.dflow DEBUG: Task 808 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 809 has memoization hash c7bc512654d8ad6e2b35e86f076f997b
parsl.dataflow.memoization INFO: Task 809 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 1038
parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0687f110-cd2d-4119-8853-f98605d07bb3_calibrate_947523_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0687f110-cd2d-4119-8853-f98605d07bb3_calibrate_947523_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/994c376f-f467-4d93-b895-fbd14bf1a35e_characterizeImage_947523_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/994c376f-f467-4d93-b895-fbd14bf1a35e_characterizeImage_947523_33.stderr
parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91a78581e860dcfa15c7dea5c605e6a0 with result from task 464
parsl.dataflow.dflow DEBUG: Task 464 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/cc64c4a8-612e-4d8c-a3ea-ee72126af340_calibrate_947418_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/cc64c4a8-612e-4d8c-a3ea-ee72126af340_calibrate_947418_43.stderr
parsl.dataflow.dflow INFO: Task 1138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99c44c3eec041290479cd367177b4900 with result from task 1138
parsl.dataflow.dflow DEBUG: Task 1138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1139 has memoization hash 9e47c85894fc05fe475fd8d81ac1c87a
parsl.dataflow.memoization INFO: Task 1139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1139 try 0 launched on executor multi with executor id 1039
parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e58ea7d9-01f5-462b-bb92-1b9fb73c1bde_calibrate_947418_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e58ea7d9-01f5-462b-bb92-1b9fb73c1bde_calibrate_947418_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4961b3a7-1c5b-4ddc-8fc8-a92af7036a0f_characterizeImage_947418_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4961b3a7-1c5b-4ddc-8fc8-a92af7036a0f_characterizeImage_947418_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:03 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c736ecad6e55a626b18122b4d8061d04 with result from task 610
parsl.dataflow.dflow DEBUG: Task 610 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 611 has memoization hash c81b70c6c982233e4fead57ec531c12f
parsl.dataflow.memoization INFO: Task 611 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 1040
parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f6d8b962-e523-4c12-9674-be8d9b753118_calibrate_947526_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f6d8b962-e523-4c12-9674-be8d9b753118_calibrate_947526_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/980ac996-1a2f-4304-adcc-88ffd72a0b75_characterizeImage_947526_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/980ac996-1a2f-4304-adcc-88ffd72a0b75_characterizeImage_947526_62.stderr
parsl.dataflow.dflow INFO: Task 1285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a65ec887d6aba8f25e73f0209635b9a with result from task 1285
parsl.dataflow.dflow DEBUG: Task 1285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1286 has memoization hash 186d99b783f5d78cab95f4240652a1f7
parsl.dataflow.memoization INFO: Task 1286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1286 try 0 launched on executor multi with executor id 1041
parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a1a86e92-11de-4494-9465-c17ac00945f4_calibrate_947526_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a1a86e92-11de-4494-9465-c17ac00945f4_calibrate_947526_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8b840617-2588-40e3-b751-c50f6caa858c_characterizeImage_947526_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8b840617-2588-40e3-b751-c50f6caa858c_characterizeImage_947526_1.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 907895f9f62563fadf33882a6028051e with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 30c28cb3523ca12ca938bb415834e379
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 1042
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2b3b6618-87eb-4a8b-8c2b-80e8c152d6e8_calibrate_947523_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2b3b6618-87eb-4a8b-8c2b-80e8c152d6e8_calibrate_947523_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ab604a63-f070-4bd5-977f-163d0c0c2b0a_characterizeImage_947523_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ab604a63-f070-4bd5-977f-163d0c0c2b0a_characterizeImage_947523_32.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d051c6579ae10e8048fd3b06191d8123 with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 060bf9860aa707f7d83999d645143861
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 1043
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/56bc831b-0bff-4140-a429-8ea8782c9167_calibrate_947526_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/56bc831b-0bff-4140-a429-8ea8782c9167_calibrate_947526_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2709140e-7397-4b99-a802-6f4c132f6a7b_characterizeImage_947526_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2709140e-7397-4b99-a802-6f4c132f6a7b_characterizeImage_947526_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:04 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c11dedd9148132f67961447281deb9f with result from task 215
parsl.dataflow.dflow DEBUG: Task 215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d6bea60c-aea1-4507-a42b-977e062c3b2f_calibrate_947418_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d6bea60c-aea1-4507-a42b-977e062c3b2f_calibrate_947418_59.stderr
parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aca31465f37ca2dfc2ba70cbda031fae with result from task 839
parsl.dataflow.dflow DEBUG: Task 839 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4a5ddfc7-31a6-4f23-90a4-f4cb715dd209_calibrate_947420_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4a5ddfc7-31a6-4f23-90a4-f4cb715dd209_calibrate_947420_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:04 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06d00f857c723ccb1f300de7cf2bbee2 with result from task 623
parsl.dataflow.dflow DEBUG: Task 623 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e8a938c5-d8b1-48bc-8f93-919da3c65898_calibrate_947523_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e8a938c5-d8b1-48bc-8f93-919da3c65898_calibrate_947523_55.stderr
parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cd19485f6c50a425cfd1a6217a5dcd1 with result from task 643
parsl.dataflow.dflow DEBUG: Task 643 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 72d862c35d6e458fe01e00e0a8e6476e
parsl.dataflow.memoization INFO: Task 644 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 1044
parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e167c6b1-59b5-4874-9cce-8d5611454024_calibrate_947524_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e167c6b1-59b5-4874-9cce-8d5611454024_calibrate_947524_49.stderr
parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ae18e70c-28f0-4702-9ca4-21558ab40af9_characterizeImage_947524_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ae18e70c-28f0-4702-9ca4-21558ab40af9_characterizeImage_947524_49.stderr
parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a538b3c16c650bf2398f5d3502ed5a64 with result from task 914
parsl.dataflow.dflow DEBUG: Task 914 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/70c1b9fe-67e1-46f4-881b-f126bc0bc6e0_calibrate_947418_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/70c1b9fe-67e1-46f4-881b-f126bc0bc6e0_calibrate_947418_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:05 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f914d928e124df05103c576d1bf7dadf with result from task 590
parsl.dataflow.dflow DEBUG: Task 590 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/795e3613-c0bb-4e76-8d40-0582a7694b2d_calibrate_947422_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/795e3613-c0bb-4e76-8d40-0582a7694b2d_calibrate_947422_47.stderr
parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 202d8dcaacb3a50a8b5f819840c0cf3d with result from task 385
parsl.dataflow.dflow DEBUG: Task 385 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 386 has memoization hash ca1f35fa4f52fa6318c32c5bf8f92839
parsl.dataflow.memoization INFO: Task 386 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f239c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 1045
parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/573593c0-4ade-4dde-aab2-19ec4f60916c_calibrate_947422_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/573593c0-4ade-4dde-aab2-19ec4f60916c_calibrate_947422_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/cbbaf19e-5d13-4bd4-9fe0-fdd8749c8c28_characterizeImage_947422_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/cbbaf19e-5d13-4bd4-9fe0-fdd8749c8c28_characterizeImage_947422_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:05 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33b2663c71ed9cb5a8784b3f102a13d2 with result from task 865
parsl.dataflow.dflow DEBUG: Task 865 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 866 has memoization hash b56dd1643b50b37dee89f26206a50f50
parsl.dataflow.memoization INFO: Task 866 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 1046
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/035453b7-2a9a-4988-9c55-f5e6ff9ed280_calibrate_947526_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/035453b7-2a9a-4988-9c55-f5e6ff9ed280_calibrate_947526_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/a23af7c2-c24e-4df9-83f3-4a35ecff1ece_characterizeImage_947526_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/a23af7c2-c24e-4df9-83f3-4a35ecff1ece_characterizeImage_947526_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:05 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 706006c067df72bc2ef9fbe4fa80e3fd with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash c820a42dc50661a1241ea67898fd0f5f
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa464040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 1047
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d2f32d26-4a57-4e46-b4dd-3fa94447361b_calibrate_947524_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d2f32d26-4a57-4e46-b4dd-3fa94447361b_calibrate_947524_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cedd7a7a-ed9a-4390-9bfb-cbaa5c3907be_characterizeImage_947524_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cedd7a7a-ed9a-4390-9bfb-cbaa5c3907be_characterizeImage_947524_31.stderr
parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd66ca082839f04a1b7fd5bc7a590e4 with result from task 508
parsl.dataflow.dflow DEBUG: Task 508 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 62a348973cbac841570a2d5fbeee04a4
parsl.dataflow.memoization INFO: Task 509 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 1048
parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ee67882-1098-4530-a027-d2cb326c8141_calibrate_947526_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ee67882-1098-4530-a027-d2cb326c8141_calibrate_947526_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/9e04b372-407d-417d-8711-2e4d7912590e_characterizeImage_947526_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/9e04b372-407d-417d-8711-2e4d7912590e_characterizeImage_947526_37.stderr
parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae560dab7dbbdb2c79b17ef40028d640 with result from task 1072
parsl.dataflow.dflow DEBUG: Task 1072 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 2c43279ffb5e28aa21467ef56ca25a80
parsl.dataflow.memoization INFO: Task 1073 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 1049
parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f16312da-0ef1-4432-b252-349e9f9c2821_calibrate_947524_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f16312da-0ef1-4432-b252-349e9f9c2821_calibrate_947524_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8200a910-e4eb-4a0f-a8fe-6829c982221f_characterizeImage_947524_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8200a910-e4eb-4a0f-a8fe-6829c982221f_characterizeImage_947524_22.stderr
parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85a5fde0ec29070cb72a2266f22086e9 with result from task 604
parsl.dataflow.dflow DEBUG: Task 604 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 3e00fa91acd15c71c84744982525360f
parsl.dataflow.memoization INFO: Task 605 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 1050
parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae73baa-69ec-4166-836a-cc3bb21385e9_calibrate_947524_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae73baa-69ec-4166-836a-cc3bb21385e9_calibrate_947524_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7230a5cd-fee0-4552-8ddc-905fc96f67a7_characterizeImage_947524_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7230a5cd-fee0-4552-8ddc-905fc96f67a7_characterizeImage_947524_62.stderr
parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac2a324dfc6ad6720b8894dc7178fbe0 with result from task 694
parsl.dataflow.dflow DEBUG: Task 694 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 695 has memoization hash f85b5c28dc41ef6e2019d689c54c20fc
parsl.dataflow.memoization INFO: Task 695 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 1051
parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b8435979-a08f-4dd7-b984-2c0ee5a4c4e7_calibrate_947422_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b8435979-a08f-4dd7-b984-2c0ee5a4c4e7_calibrate_947422_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/46a2ca61-b0b7-4eb2-89b6-75fd8975697e_characterizeImage_947422_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/46a2ca61-b0b7-4eb2-89b6-75fd8975697e_characterizeImage_947422_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:05 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 203582603327a11ddb55d0d2ed2addf7 with result from task 892
parsl.dataflow.dflow DEBUG: Task 892 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 42963f90099c19be1238597e294761c8
parsl.dataflow.memoization INFO: Task 893 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 1052
parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/02dba8bf-b8b0-41cc-a7d9-4faded03d67c_calibrate_947422_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/02dba8bf-b8b0-41cc-a7d9-4faded03d67c_calibrate_947422_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4fb07e3c-4a52-4150-82a2-d1b301399ec7_characterizeImage_947422_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/4fb07e3c-4a52-4150-82a2-d1b301399ec7_characterizeImage_947422_14.stderr
parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5468a086b7bf2beb9931456f75050c15 with result from task 815
parsl.dataflow.dflow DEBUG: Task 815 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/bd0c5fa9-8188-4554-97db-ac29c3dea1bb_calibrate_947418_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/bd0c5fa9-8188-4554-97db-ac29c3dea1bb_calibrate_947418_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:06 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 391a21e3f50bde5397e56ee31d717367 with result from task 1057
parsl.dataflow.dflow DEBUG: Task 1057 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash e4d73ff6fe88c044759ccd261578d7fd
parsl.dataflow.memoization INFO: Task 1058 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1053
parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/55054fed-ffde-488b-b4ae-fc7e8aa815e5_calibrate_947528_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/55054fed-ffde-488b-b4ae-fc7e8aa815e5_calibrate_947528_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/659e3651-f548-46d5-85df-39fc19d50ba5_characterizeImage_947528_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/659e3651-f548-46d5-85df-39fc19d50ba5_characterizeImage_947528_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:06 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c976d23c9038ed9f51781cc2f6acdf60 with result from task 1075
parsl.dataflow.dflow DEBUG: Task 1075 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash a1458753c0e9c43bfbdf87bce9cc6af8
parsl.dataflow.memoization INFO: Task 1076 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c34d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1054
parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9213d8f4-df36-44e9-bd79-ec4be670f505_calibrate_947422_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9213d8f4-df36-44e9-bd79-ec4be670f505_calibrate_947422_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/85722506-226c-48c8-a828-d31adcbf2f0c_characterizeImage_947422_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/85722506-226c-48c8-a828-d31adcbf2f0c_characterizeImage_947422_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:06 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:06 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8c2eac24201a5e4656a8414fbac3265 with result from task 985
parsl.dataflow.dflow DEBUG: Task 985 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 986 has memoization hash 366078ce6a319e2bb7c9bdcdb3da61bf
parsl.dataflow.memoization INFO: Task 986 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 1055
parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b8d5f24-1c73-4ed0-bfee-ab8412b94ecf_calibrate_947523_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b8d5f24-1c73-4ed0-bfee-ab8412b94ecf_calibrate_947523_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/285dc637-1141-49ec-a71e-01a5ca6f9183_characterizeImage_947523_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/285dc637-1141-49ec-a71e-01a5ca6f9183_characterizeImage_947523_29.stderr
parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ebfc3542540835b982448e077e3ae58 with result from task 625
parsl.dataflow.dflow DEBUG: Task 625 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 626 has memoization hash ee3d7e5d145281263b39b384b06f9f09
parsl.dataflow.memoization INFO: Task 626 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 1056
parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bf17c699-c9a8-4830-b831-8a3d1b621df5_calibrate_947420_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bf17c699-c9a8-4830-b831-8a3d1b621df5_calibrate_947420_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/14e3fd95-1388-48c0-b8d8-430e411162ab_characterizeImage_947420_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/14e3fd95-1388-48c0-b8d8-430e411162ab_characterizeImage_947420_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:06 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16eb1e5119d25e127a5fdac05745ed47 with result from task 742
parsl.dataflow.dflow DEBUG: Task 742 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 743 has memoization hash ae1d244b425b2c681eea7213a49660b9
parsl.dataflow.memoization INFO: Task 743 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 1057
parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8454de94-ae04-4bf0-a681-b26726fd00bc_calibrate_947524_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8454de94-ae04-4bf0-a681-b26726fd00bc_calibrate_947524_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/866191ff-0656-43ba-853f-74227b19ceb4_characterizeImage_947524_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/866191ff-0656-43ba-853f-74227b19ceb4_characterizeImage_947524_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:07 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f98c7437e74fe0c4c39690e3b27d7c4 with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/18241bb4-40e0-4a1f-a618-8f915ded38fc_calibrate_947418_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/18241bb4-40e0-4a1f-a618-8f915ded38fc_calibrate_947418_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:07 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:07 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:07 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:07 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc6f53b6c01809bc99896b6bfef95e1b with result from task 428
parsl.dataflow.dflow DEBUG: Task 428 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ea8df07f-230f-481a-8fec-0c90db5f1885_calibrate_947420_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ea8df07f-230f-481a-8fec-0c90db5f1885_calibrate_947420_3.stderr
parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34728f90581cf55b1d4961bc2913bc7d with result from task 574
parsl.dataflow.dflow DEBUG: Task 574 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 00ee52ea7a0db1f95cea1bbd8c22748f
parsl.dataflow.memoization INFO: Task 575 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dccea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 1058
parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fd50a89f-d1b6-4293-b0dc-a7608606ddb1_calibrate_947524_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fd50a89f-d1b6-4293-b0dc-a7608606ddb1_calibrate_947524_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/371ef057-2bf2-4296-bc8a-6cb1ce7b5ce0_characterizeImage_947524_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/371ef057-2bf2-4296-bc8a-6cb1ce7b5ce0_characterizeImage_947524_29.stderr
parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46fdb20551de014880f21721c811c6ce with result from task 634
parsl.dataflow.dflow DEBUG: Task 634 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 635 has memoization hash b4059d85b6031b877050ee59b059ea96
parsl.dataflow.memoization INFO: Task 635 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 1059
parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/217b5c10-c426-4bb6-82ad-898a624c61bc_calibrate_947418_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/217b5c10-c426-4bb6-82ad-898a624c61bc_calibrate_947418_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c7364642-399e-49e0-8efe-d8256646f47c_characterizeImage_947418_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/c7364642-399e-49e0-8efe-d8256646f47c_characterizeImage_947418_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry deebd9ac87f689aa6d56671d51778263 with result from task 484
parsl.dataflow.dflow DEBUG: Task 484 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 6a8a11ecafe1b0301c753e00fc13639e
parsl.dataflow.memoization INFO: Task 485 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d86160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 1060
parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/62114cc7-f8c0-46b5-a30a-1cc4396971c4_calibrate_947528_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/62114cc7-f8c0-46b5-a30a-1cc4396971c4_calibrate_947528_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/75ec388f-6f50-4281-878a-8c75eb1ca030_characterizeImage_947528_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/75ec388f-6f50-4281-878a-8c75eb1ca030_characterizeImage_947528_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b192ef54f393445ec3a7327718e948ac with result from task 1045
parsl.dataflow.dflow DEBUG: Task 1045 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 1e41bce103f44902a4bce7b190bd2381
parsl.dataflow.memoization INFO: Task 1046 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 1061
parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8c070fcb-74bc-4984-b9af-105c15defad6_calibrate_947422_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8c070fcb-74bc-4984-b9af-105c15defad6_calibrate_947422_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/86663901-8359-428a-9af5-0b1af26b7256_characterizeImage_947422_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/86663901-8359-428a-9af5-0b1af26b7256_characterizeImage_947422_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:08 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23f5dbcdcef49be29ada7c75315af027 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash e264bcc4bbde49a67e151e3713db4822
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 1062
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a32f9ab-126f-4ff0-972c-6f9093c2eb00_calibrate_947523_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a32f9ab-126f-4ff0-972c-6f9093c2eb00_calibrate_947523_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ec696ee2-d0ae-488b-8d00-302bc06926b6_characterizeImage_947523_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/ec696ee2-d0ae-488b-8d00-302bc06926b6_characterizeImage_947523_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 226dbec23957610548b809da19d225e4 with result from task 491
parsl.dataflow.dflow DEBUG: Task 491 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/28e16602-4adc-4a00-9aed-b1ab851620b6_calibrate_947523_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/28e16602-4adc-4a00-9aed-b1ab851620b6_calibrate_947523_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e4e85ee64681f47d8cf34d09c2c2476 with result from task 616
parsl.dataflow.dflow DEBUG: Task 616 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 5f29c3b6889655b1df5ec2c46930430f
parsl.dataflow.memoization INFO: Task 617 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dced40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 1063
parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a82ef996-a77e-41fa-bf38-e991e570cb82_calibrate_947526_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a82ef996-a77e-41fa-bf38-e991e570cb82_calibrate_947526_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/58b6e822-5a51-4ff8-af2d-980872eab0fd_characterizeImage_947526_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/58b6e822-5a51-4ff8-af2d-980872eab0fd_characterizeImage_947526_9.stderr
parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59d895eb15b5b61e606d481475d5493a with result from task 878
parsl.dataflow.dflow DEBUG: Task 878 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f3ac932e-321a-4568-9312-8c4e38493000_calibrate_947418_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f3ac932e-321a-4568-9312-8c4e38493000_calibrate_947418_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c62e7af0521b61c72906addb7f3077f with result from task 203
parsl.dataflow.dflow DEBUG: Task 203 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/3ec6246c-e7bb-4bea-99cc-4287ca279156_calibrate_947422_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/3ec6246c-e7bb-4bea-99cc-4287ca279156_calibrate_947422_59.stderr
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20ce315cfae902bb6dd078db69cc62fd with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 6b8f23aba2f763f347cacb3f4a7c465b
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 1064
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e3e91131-651d-4d45-9b4a-d77107f709f8_calibrate_947523_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e3e91131-651d-4d45-9b4a-d77107f709f8_calibrate_947523_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/af825a7f-a7be-4e6b-b394-514fa0162b28_characterizeImage_947523_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/af825a7f-a7be-4e6b-b394-514fa0162b28_characterizeImage_947523_16.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62e924eb7c241e5a30f062d8b4feb5d0 with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 0dbd07a3560fd0f311c332ef256b71a2
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 1065
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/56aba7cf-65a4-46ed-907c-cc781c72cb62_calibrate_947418_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/56aba7cf-65a4-46ed-907c-cc781c72cb62_calibrate_947418_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a28b34d5-d261-4aeb-909b-e4201fdb9a47_characterizeImage_947418_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a28b34d5-d261-4aeb-909b-e4201fdb9a47_characterizeImage_947418_53.stderr
parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb596d8fa8d74d35cb1e2b280ca390e9 with result from task 781
parsl.dataflow.dflow DEBUG: Task 781 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 782 has memoization hash b720215cc4ca6915481abbaefeedf1b3
parsl.dataflow.memoization INFO: Task 782 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 1066
parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/1fb7ac68-db64-4a7f-a64b-ab54c23ebb04_calibrate_947422_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/1fb7ac68-db64-4a7f-a64b-ab54c23ebb04_calibrate_947422_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6dc89df7-22a1-43a8-b44f-a1d88965a934_characterizeImage_947422_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6dc89df7-22a1-43a8-b44f-a1d88965a934_characterizeImage_947422_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 135d9c29b27eaa0def58668460e00577 with result from task 334
parsl.dataflow.dflow DEBUG: Task 334 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 2de4ab23a83be1bc77cd8fcf60cd3b4d
parsl.dataflow.memoization INFO: Task 335 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 1067
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/92cdb360-9aeb-4a72-b12a-7d8876145a7c_calibrate_947524_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/92cdb360-9aeb-4a72-b12a-7d8876145a7c_calibrate_947524_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/97091f58-64f8-4dd4-afda-997df0c34393_characterizeImage_947524_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/97091f58-64f8-4dd4-afda-997df0c34393_characterizeImage_947524_34.stderr
parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10243591bb23f58a3f07330792bd1c1c with result from task 329
parsl.dataflow.dflow DEBUG: Task 329 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ded2a22e-0dce-4109-b7c9-36c8fb78ac51_calibrate_947418_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ded2a22e-0dce-4109-b7c9-36c8fb78ac51_calibrate_947418_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:09 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4d57051018bcf6a2de17b7d068c4a69 with result from task 728
parsl.dataflow.dflow DEBUG: Task 728 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e429858f-b901-4c29-8700-0da3ba1a3a82_calibrate_947418_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e429858f-b901-4c29-8700-0da3ba1a3a82_calibrate_947418_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20e797b4dd30bc86aafba4b640a09978 with result from task 601
parsl.dataflow.dflow DEBUG: Task 601 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 38963556736b31d54dbe42fb348a306a
parsl.dataflow.memoization INFO: Task 602 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 1068
parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f07fef78-0437-4016-a574-31e1ec3bf024_calibrate_947526_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f07fef78-0437-4016-a574-31e1ec3bf024_calibrate_947526_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ea5fd53e-4c7c-4683-838c-9fdc0bee02a6_characterizeImage_947526_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ea5fd53e-4c7c-4683-838c-9fdc0bee02a6_characterizeImage_947526_27.stderr
parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9de9b6b77ff3cf1320d9ba584551163 with result from task 442
parsl.dataflow.dflow DEBUG: Task 442 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 443 has memoization hash c577a869b6d71d3801cf966dc5a7edc4
parsl.dataflow.memoization INFO: Task 443 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 1069
parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d402039c-4688-46b4-919a-c65701699f21_calibrate_947528_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d402039c-4688-46b4-919a-c65701699f21_calibrate_947528_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2bc03435-3008-4dce-aae6-802c998ee50b_characterizeImage_947528_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2bc03435-3008-4dce-aae6-802c998ee50b_characterizeImage_947528_21.stderr
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64f129cca316e8c8b77408b741cc047d with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 91b73f75083f57cae8c1fc3b8d6c9564
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 1070
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/dd5497f1-2aa3-4458-b2eb-6ad8c50b60a9_calibrate_947420_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/dd5497f1-2aa3-4458-b2eb-6ad8c50b60a9_calibrate_947420_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/47ca7f3c-ad65-4a75-a8eb-625c16822815_characterizeImage_947420_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/47ca7f3c-ad65-4a75-a8eb-625c16822815_characterizeImage_947420_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2c5dbd1374f692f5e234416f58644f2 with result from task 247
parsl.dataflow.dflow DEBUG: Task 247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 35d1df0161341e08bd9728298769d639
parsl.dataflow.memoization INFO: Task 248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 1071
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/c90e5d0a-9bd3-4496-845d-f37c1976eaec_calibrate_947420_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/c90e5d0a-9bd3-4496-845d-f37c1976eaec_calibrate_947420_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/56b97cb6-b815-466e-b5ce-8aafbc019464_characterizeImage_947420_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/56b97cb6-b815-466e-b5ce-8aafbc019464_characterizeImage_947420_23.stderr
parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fde4021bddcb625558adef32e06c45f4 with result from task 931
parsl.dataflow.dflow DEBUG: Task 931 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 932 has memoization hash c5909e0509a76147e6ebfeb055280790
parsl.dataflow.memoization INFO: Task 932 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 1072
parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ca29e3a5-f9a1-4dc3-aead-7140cec38064_calibrate_947523_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ca29e3a5-f9a1-4dc3-aead-7140cec38064_calibrate_947523_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8dcee7ba-041f-47ed-acab-171a0a118444_characterizeImage_947523_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8dcee7ba-041f-47ed-acab-171a0a118444_characterizeImage_947523_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42d2f384e86c8425b11915cafb852a29 with result from task 820
parsl.dataflow.dflow DEBUG: Task 820 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 821 has memoization hash c1c946856d36ca08a0157ad395925973
parsl.dataflow.memoization INFO: Task 821 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d082c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 1073
parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/58cb3447-fc8f-42b1-a6d9-63306a2173c9_calibrate_947526_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/58cb3447-fc8f-42b1-a6d9-63306a2173c9_calibrate_947526_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/119ec7b9-5a77-4f17-8812-0cdd0a513aad_characterizeImage_947526_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/119ec7b9-5a77-4f17-8812-0cdd0a513aad_characterizeImage_947526_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d927821c0e72d930dd0a0c149a8f59d with result from task 640
parsl.dataflow.dflow DEBUG: Task 640 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 641 has memoization hash c0c0684c0c877a89d993ea62160c6dcc
parsl.dataflow.memoization INFO: Task 641 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 1074
parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5efd9047-a85e-4137-98e7-f7cbb8609c60_calibrate_947528_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5efd9047-a85e-4137-98e7-f7cbb8609c60_calibrate_947528_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/77031e25-b203-48a0-846d-e28aa37b026b_characterizeImage_947528_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/77031e25-b203-48a0-846d-e28aa37b026b_characterizeImage_947528_33.stderr
parsl.dataflow.dflow INFO: Task 1174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 984efc46a630b671f16f9c05b78f9f5a with result from task 1174
parsl.dataflow.dflow DEBUG: Task 1174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1175 has memoization hash 0f3e0aef379f7354f34f5e80a82e9ebb
parsl.dataflow.memoization INFO: Task 1175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1175 try 0 launched on executor multi with executor id 1075
parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/84296262-942a-4563-a7f6-c6276cc2dc74_calibrate_947528_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/84296262-942a-4563-a7f6-c6276cc2dc74_calibrate_947528_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ec32677c-d603-4688-9fc4-6389e72e311f_characterizeImage_947528_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ec32677c-d603-4688-9fc4-6389e72e311f_characterizeImage_947528_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:10 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8885b4cec274fad242485ed83b06ff with result from task 290
parsl.dataflow.dflow DEBUG: Task 290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/a3ea8622-e4a0-444e-b570-00c68bc82a69_calibrate_947420_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/a3ea8622-e4a0-444e-b570-00c68bc82a69_calibrate_947420_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:11 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4f5929578035551ef9e37c08683c07 with result from task 211
parsl.dataflow.dflow DEBUG: Task 211 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 212 has memoization hash d82c0d7ed7346a41c28d909efc7b77b1
parsl.dataflow.memoization INFO: Task 212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 1076
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5abdbbb4-89b8-443e-aa20-4c3e22070600_calibrate_947418_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5abdbbb4-89b8-443e-aa20-4c3e22070600_calibrate_947418_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/63460f60-c94f-4dee-83b6-983970b33eb4_characterizeImage_947418_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/63460f60-c94f-4dee-83b6-983970b33eb4_characterizeImage_947418_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:11 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b117767e06d2fe762122b51c765fc48 with result from task 943
parsl.dataflow.dflow DEBUG: Task 943 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 0fd2368f8eacd3da5a3fa9bb10b215a4
parsl.dataflow.memoization INFO: Task 944 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 1077
parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/4df6149b-8c27-47ad-bf4a-ef98ecb64581_calibrate_947528_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/4df6149b-8c27-47ad-bf4a-ef98ecb64581_calibrate_947528_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/c1f70494-9dc5-4f90-8691-048674f13407_characterizeImage_947528_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/c1f70494-9dc5-4f90-8691-048674f13407_characterizeImage_947528_51.stderr
parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry decd62d1ff6f00ef7fb36f0b1e6c57c2 with result from task 331
parsl.dataflow.dflow DEBUG: Task 331 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 753dacabc6ff7e2a3d5d72d10a191050
parsl.dataflow.memoization INFO: Task 332 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 1078
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/320759db-cbc3-4f26-865a-66fd3fb4a162_calibrate_947422_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/320759db-cbc3-4f26-865a-66fd3fb4a162_calibrate_947422_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebf3b7b5-278a-4b02-9b07-9dab6bb345a5_characterizeImage_947422_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ebf3b7b5-278a-4b02-9b07-9dab6bb345a5_characterizeImage_947422_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:11 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47f28732d96b69964ad25a24220af972 with result from task 233
parsl.dataflow.dflow DEBUG: Task 233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2d2c706f-135b-42d5-9c0b-84536c68b73d_calibrate_947420_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2d2c706f-135b-42d5-9c0b-84536c68b73d_calibrate_947420_36.stderr
parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9765c2004f57d99add9a65ecef8a9461 with result from task 850
parsl.dataflow.dflow DEBUG: Task 850 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 140f77dc2deb961ca9644db26ad48b87
parsl.dataflow.memoization INFO: Task 851 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 1079
parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/54710d40-45dc-4192-a776-a12c66cbd770_calibrate_947523_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/54710d40-45dc-4192-a776-a12c66cbd770_calibrate_947523_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/adfe1bb8-609c-43c4-af28-9bc4d58af67f_characterizeImage_947523_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/adfe1bb8-609c-43c4-af28-9bc4d58af67f_characterizeImage_947523_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:11 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4e2ab6ca7b5b162a8523d5e5d5a3948 with result from task 904
parsl.dataflow.dflow DEBUG: Task 904 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 26efee371aad96a3294ade0927c5491f
parsl.dataflow.memoization INFO: Task 905 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 1080
parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e7fdd721-25b4-450c-ab62-2e280868c753_calibrate_947422_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e7fdd721-25b4-450c-ab62-2e280868c753_calibrate_947422_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7682c06b-84a1-42c0-b4aa-1ba6fce09ba5_characterizeImage_947422_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7682c06b-84a1-42c0-b4aa-1ba6fce09ba5_characterizeImage_947422_41.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef484a6ddcb6c05b215e89f4ff9ceb49 with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash cd529a776accb4fc52155509f2a41040
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 1081
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/237f2531-ae62-428b-b29d-18abc6068afa_calibrate_947528_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/237f2531-ae62-428b-b29d-18abc6068afa_calibrate_947528_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/396c8a57-0532-4a2b-885d-86db6f159e94_characterizeImage_947528_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/396c8a57-0532-4a2b-885d-86db6f159e94_characterizeImage_947528_6.stderr
parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f5c01d363ba84ad7ee208c79c7e3ecc with result from task 733
parsl.dataflow.dflow DEBUG: Task 733 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 1c752acec1b753c9f29431ba248c9a5c
parsl.dataflow.memoization INFO: Task 734 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 1082
parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/bc4e2906-0428-4a58-9aa3-6d822eb58837_calibrate_947524_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/bc4e2906-0428-4a58-9aa3-6d822eb58837_calibrate_947524_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8a8da536-b8d5-43e5-81ae-5c3f867dd53c_characterizeImage_947524_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8a8da536-b8d5-43e5-81ae-5c3f867dd53c_characterizeImage_947524_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:11 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4d3396e49dd6a3f2e7e1244cbce94130 with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 9661a7c1c540eabb9efbc78f63560736
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 1083
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/7307ed97-04b3-4e12-b1cb-53eb43e5b994_calibrate_947523_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/7307ed97-04b3-4e12-b1cb-53eb43e5b994_calibrate_947523_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5eaf682c-b1b1-4a67-8f8e-17acb3222e15_characterizeImage_947523_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5eaf682c-b1b1-4a67-8f8e-17acb3222e15_characterizeImage_947523_36.stderr
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cca61043a9e7b41715b3ff48b3a1f7b5 with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8840cf13-6f08-4beb-8781-47ff36837dc0_calibrate_947420_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8840cf13-6f08-4beb-8781-47ff36837dc0_calibrate_947420_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:12 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7092425514555cd0f644f3c7392d0ed with result from task 1093
parsl.dataflow.dflow DEBUG: Task 1093 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 85d979d4e39b9462dda9ce600c54f072
parsl.dataflow.memoization INFO: Task 1094 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1084
parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/6b03b0f9-df96-40d9-bac9-4ec6850419c7_calibrate_947420_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/6b03b0f9-df96-40d9-bac9-4ec6850419c7_calibrate_947420_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/adcc61b8-24de-48e8-99ea-6ddc0bbd1967_characterizeImage_947420_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/adcc61b8-24de-48e8-99ea-6ddc0bbd1967_characterizeImage_947420_54.stderr
parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f41f69d54f35414c51bdb3f7cb53408 with result from task 542
parsl.dataflow.dflow DEBUG: Task 542 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fce1d0e5-d9af-4944-ba03-69006126e684_calibrate_947418_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fce1d0e5-d9af-4944-ba03-69006126e684_calibrate_947418_9.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf1507d917f6debeda6ef688866fe18d with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash c0a14218bb2e4fdba6ab1ca6bcf49fc6
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 1085
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1019c02d-bbba-4d80-b922-9c079508a8bc_calibrate_947418_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1019c02d-bbba-4d80-b922-9c079508a8bc_calibrate_947418_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cb4f827f-9be2-480e-87e2-d6dabe86bc8c_characterizeImage_947418_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cb4f827f-9be2-480e-87e2-d6dabe86bc8c_characterizeImage_947418_21.stderr
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fdfe2b01c703530968a7bf09ede4e68 with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash a52c2451b1f759165846fa0b562db912
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 1086
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/39076b8e-9d22-40d2-886c-334870e19df7_calibrate_947418_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/39076b8e-9d22-40d2-886c-334870e19df7_calibrate_947418_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/74acadc3-9f20-4a58-b812-3361a2fa46fa_characterizeImage_947418_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/74acadc3-9f20-4a58-b812-3361a2fa46fa_characterizeImage_947418_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:12 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:12 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c057551445a9ec60f238f1ccc342dd80 with result from task 223
parsl.dataflow.dflow DEBUG: Task 223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 224 has memoization hash de3c00695bfa3631da35439774432f43
parsl.dataflow.memoization INFO: Task 224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 1087
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e6887045-cfd4-4606-852f-c3867587c46d_calibrate_947422_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e6887045-cfd4-4606-852f-c3867587c46d_calibrate_947422_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/504281c8-3610-46d5-bc41-4d4f222cc8a9_characterizeImage_947422_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/504281c8-3610-46d5-bc41-4d4f222cc8a9_characterizeImage_947422_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:13 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a5b5c575f137d0b32e42ce46b3e655e with result from task 823
parsl.dataflow.dflow DEBUG: Task 823 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 858177e26e8ec74f7734061752eb756a
parsl.dataflow.memoization INFO: Task 824 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d085e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 1088
parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/34b80b96-2409-4a52-8244-f566e070f366_calibrate_947528_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/34b80b96-2409-4a52-8244-f566e070f366_calibrate_947528_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d1cf777-3f7b-4ebf-b70c-181b84e6e156_characterizeImage_947528_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d1cf777-3f7b-4ebf-b70c-181b84e6e156_characterizeImage_947528_37.stderr
parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e2d57eabeb13e3079c39afdfab15737e with result from task 367
parsl.dataflow.dflow DEBUG: Task 367 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 110b6c753d17f3f6ce79727bd2490aa5
parsl.dataflow.memoization INFO: Task 368 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f22700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 1089
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6b60cde9-acdd-4a63-ae69-5524a9f6dd3f_calibrate_947528_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6b60cde9-acdd-4a63-ae69-5524a9f6dd3f_calibrate_947528_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/926ec619-c793-4970-9670-bd688ad779bf_characterizeImage_947528_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/926ec619-c793-4970-9670-bd688ad779bf_characterizeImage_947528_54.stderr
parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20465c79565d1cb1d621e830bd98c1c6 with result from task 526
parsl.dataflow.dflow DEBUG: Task 526 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 63fef84ba207294447588ad6677cde07
parsl.dataflow.memoization INFO: Task 527 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 1090
parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/44343698-783d-401a-bdb6-0343ed9f6e2f_calibrate_947418_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/44343698-783d-401a-bdb6-0343ed9f6e2f_calibrate_947418_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eb93afa5-c41f-4fec-b796-346f5960a388_characterizeImage_947418_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/eb93afa5-c41f-4fec-b796-346f5960a388_characterizeImage_947418_17.stderr
parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ae9581883d6120b7f7c1a12b73d8c9d with result from task 883
parsl.dataflow.dflow DEBUG: Task 883 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 5642b5421fdaa4044d9aaa1f877857f7
parsl.dataflow.memoization INFO: Task 884 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 1091
parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6aabdcc8-00c1-4a8d-89b9-e734dc3f10f9_calibrate_947528_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6aabdcc8-00c1-4a8d-89b9-e734dc3f10f9_calibrate_947528_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/deda3783-4c05-40a1-afad-9060380ee177_characterizeImage_947528_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/deda3783-4c05-40a1-afad-9060380ee177_characterizeImage_947528_18.stderr
parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1dced2c24f2463085c67dc07bbf23b4d with result from task 578
parsl.dataflow.dflow DEBUG: Task 578 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2335e514-00e5-4cd0-878b-c05d9c32c3af_calibrate_947523_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2335e514-00e5-4cd0-878b-c05d9c32c3af_calibrate_947523_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:13 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d107adab47c7d77d4f0bb006f9c3b85 with result from task 269
parsl.dataflow.dflow DEBUG: Task 269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/45f69b97-7827-4b72-8783-1b142aaecf51_calibrate_947528_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/45f69b97-7827-4b72-8783-1b142aaecf51_calibrate_947528_31.stderr
parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e61774dec3f5c020996a73ae30bbb1f8 with result from task 812
parsl.dataflow.dflow DEBUG: Task 812 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/20a32d75-caf4-4680-bd0d-2dd97d117cc6_calibrate_947418_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/20a32d75-caf4-4680-bd0d-2dd97d117cc6_calibrate_947418_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:14 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42184812652284d6751eb214d5b2539a with result from task 805
parsl.dataflow.dflow DEBUG: Task 805 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 57c0683d0c860ffc322cd1f1bd344496
parsl.dataflow.memoization INFO: Task 806 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 1092
parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b6fddb5a-eccd-4bee-8374-15211bbeeb1d_calibrate_947528_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b6fddb5a-eccd-4bee-8374-15211bbeeb1d_calibrate_947528_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/50b857ff-8b99-48bc-9216-7ea131e5413c_characterizeImage_947528_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/50b857ff-8b99-48bc-9216-7ea131e5413c_characterizeImage_947528_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:14 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a8e3c2325174dde0fd7c38430b782b61 with result from task 475
parsl.dataflow.dflow DEBUG: Task 475 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 52c4bfce8bb97ce95388bd71210a7a23
parsl.dataflow.memoization INFO: Task 476 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d85800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 1093
parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/cf86ca05-2bff-4b71-8cdd-ce531580cb39_calibrate_947526_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/cf86ca05-2bff-4b71-8cdd-ce531580cb39_calibrate_947526_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fa768a72-23b4-4b72-87fe-db12ca0e200b_characterizeImage_947526_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fa768a72-23b4-4b72-87fe-db12ca0e200b_characterizeImage_947526_11.stderr
parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99ac2eb1e4149f21337f662d748198ed with result from task 488
parsl.dataflow.dflow DEBUG: Task 488 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e89a2aad-cb28-4ec9-b7e8-7c1dce5a5c49_calibrate_947422_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e89a2aad-cb28-4ec9-b7e8-7c1dce5a5c49_calibrate_947422_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:14 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dba0f1032da4ff63056930f9b04cd2c6 with result from task 545
parsl.dataflow.dflow DEBUG: Task 545 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0e2dc453-2881-4d38-9b38-1e4cd7f6a494_calibrate_947418_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0e2dc453-2881-4d38-9b38-1e4cd7f6a494_calibrate_947418_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:15 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 795cb9a82decc6d6d76f9ad7a8566f2c with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash ebcf9323595f1a39180dbc63b0083e07
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e949a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 1094
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fb6daf65-b1bf-405e-91c0-21192091e7ee_calibrate_947418_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fb6daf65-b1bf-405e-91c0-21192091e7ee_calibrate_947418_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4b2e36e6-3f51-4499-9aa0-43b7de685f57_characterizeImage_947418_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/4b2e36e6-3f51-4499-9aa0-43b7de685f57_characterizeImage_947418_39.stderr
parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9c253c16b8d0341e3f5b3ad6301496f with result from task 382
parsl.dataflow.dflow DEBUG: Task 382 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 2e089b94b68be68bcc3b26cec5ed356b
parsl.dataflow.memoization INFO: Task 383 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f236a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 1095
parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/31458834-a4d2-4e91-bd2d-a5c5bb18556f_calibrate_947420_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/31458834-a4d2-4e91-bd2d-a5c5bb18556f_calibrate_947420_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/40e934cf-a02a-4b51-988b-e31872a1f52a_characterizeImage_947420_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/40e934cf-a02a-4b51-988b-e31872a1f52a_characterizeImage_947420_24.stderr
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d3af897781808f4b5e2d80e3bbaed56 with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/53c31ddc-3112-4557-bb44-a31463ac05b0_calibrate_947418_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/53c31ddc-3112-4557-bb44-a31463ac05b0_calibrate_947418_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:15 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d4669c1cf9065011d5430679c69db30 with result from task 596
parsl.dataflow.dflow DEBUG: Task 596 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8d6731a2-1877-42de-9c5d-511e06a55711_calibrate_947422_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8d6731a2-1877-42de-9c5d-511e06a55711_calibrate_947422_53.stderr
parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec4073920be06485c3a2ef9607beaa6 with result from task 359
parsl.dataflow.dflow DEBUG: Task 359 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0aef1900-0b49-4939-a5cd-e1cd5a10f446_calibrate_947523_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0aef1900-0b49-4939-a5cd-e1cd5a10f446_calibrate_947523_42.stderr
parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 351ff48daf46919e3e633065cddc92e8 with result from task 685
parsl.dataflow.dflow DEBUG: Task 685 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 7fc15fb2841889351edde1e29094c640
parsl.dataflow.memoization INFO: Task 686 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 1096
parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fdc83cf9-ef90-417f-a99b-6e467bff11e0_calibrate_947418_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fdc83cf9-ef90-417f-a99b-6e467bff11e0_calibrate_947418_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/141242a6-4cd0-4157-93d6-8eed67302f9c_characterizeImage_947418_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/141242a6-4cd0-4157-93d6-8eed67302f9c_characterizeImage_947418_31.stderr
parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77ec0113cebca15d7743be7dcc28dbdb with result from task 394
parsl.dataflow.dflow DEBUG: Task 394 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 61afa25196d2ff80c59fdaf73bcc2703
parsl.dataflow.memoization INFO: Task 395 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 1097
parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/98bc0e40-dc29-4ef0-9f33-2db0ab702fb9_calibrate_947418_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/98bc0e40-dc29-4ef0-9f33-2db0ab702fb9_calibrate_947418_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/02196db1-baa5-415e-8637-dc4fa425611d_characterizeImage_947418_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/02196db1-baa5-415e-8637-dc4fa425611d_characterizeImage_947418_42.stderr
parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b3e8df186814cd6d7eb17c762a2417e with result from task 889
parsl.dataflow.dflow DEBUG: Task 889 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 15f3abf2b01794f68b15f624505855da
parsl.dataflow.memoization INFO: Task 890 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 1098
parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d1f00a36-3b47-4441-bc8b-0f97aed5531c_calibrate_947526_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d1f00a36-3b47-4441-bc8b-0f97aed5531c_calibrate_947526_33.stderr
parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/0a13a308-5862-4faa-994b-096b32579127_characterizeImage_947526_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/0a13a308-5862-4faa-994b-096b32579127_characterizeImage_947526_33.stderr
parsl.dataflow.dflow INFO: Task 1171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2823ea96bbcac6c6b0f3ba4d4ecdee37 with result from task 1171
parsl.dataflow.dflow DEBUG: Task 1171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1172 has memoization hash c82ad4a0c466ab109d1b7f99c2da46f6
parsl.dataflow.memoization INFO: Task 1172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:16 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Parsl task 1172 try 0 launched on executor multi with executor id 1099
parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7cc661b7-de1a-4ba4-9290-ac715975a5b4_calibrate_947526_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7cc661b7-de1a-4ba4-9290-ac715975a5b4_calibrate_947526_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/98638c71-64f9-4b2e-b85c-1f112c2b3ed6_characterizeImage_947526_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/98638c71-64f9-4b2e-b85c-1f112c2b3ed6_characterizeImage_947526_43.stderr
parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46e6a47c8b1d50202aaa99fff5fb2645 with result from task 242
parsl.dataflow.dflow DEBUG: Task 242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/599f8675-1617-45d5-b4a1-c5e6ae0ecf31_calibrate_947420_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/599f8675-1617-45d5-b4a1-c5e6ae0ecf31_calibrate_947420_46.stderr
parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a330421201195b0ee04512ce3eabcd1 with result from task 761
parsl.dataflow.dflow DEBUG: Task 761 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/efb615bb-f462-4dfe-a9f3-00faa2e1a60f_calibrate_947420_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/efb615bb-f462-4dfe-a9f3-00faa2e1a60f_calibrate_947420_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:16 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12257bdaf3ca3c958bb478aa16e7cb49 with result from task 1111
parsl.dataflow.dflow DEBUG: Task 1111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 13eee97f0cf99a6ec9a6d02b87fc41e9
parsl.dataflow.memoization INFO: Task 1112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1100
parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5d05ffca-9f24-4442-8cb8-371d68adad7d_calibrate_947418_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5d05ffca-9f24-4442-8cb8-371d68adad7d_calibrate_947418_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2089d1d5-aac9-407b-b666-9b5d0b527957_characterizeImage_947418_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/2089d1d5-aac9-407b-b666-9b5d0b527957_characterizeImage_947418_50.stderr
parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21da732757a7b391975bb42e29e6d740 with result from task 661
parsl.dataflow.dflow DEBUG: Task 661 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 8f1f7c3178ac63df94710c875756d01a
parsl.dataflow.memoization INFO: Task 662 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 1101
parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6f793069-dda6-4a10-94c9-8c8b50b464ea_calibrate_947526_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6f793069-dda6-4a10-94c9-8c8b50b464ea_calibrate_947526_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/753e4f36-5215-4490-bb7c-6456e22e73f8_characterizeImage_947526_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/753e4f36-5215-4490-bb7c-6456e22e73f8_characterizeImage_947526_8.stderr
parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab1ebe0d3f7d631800bceb781f03e8f0 with result from task 389
parsl.dataflow.dflow DEBUG: Task 389 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/acde92fd-7067-4638-9f8e-c7e663ba63c3_calibrate_947420_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/acde92fd-7067-4638-9f8e-c7e663ba63c3_calibrate_947420_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:16 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89f1973f4e5820a84f0488a40098584a with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 01204d060f8c1561a9f6172f74b42656
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 1102
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f264cca0-504d-47c3-bb57-f49d3e160664_calibrate_947523_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f264cca0-504d-47c3-bb57-f49d3e160664_calibrate_947523_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/dc228e9b-39ea-47af-9780-f5050875104a_characterizeImage_947523_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/dc228e9b-39ea-47af-9780-f5050875104a_characterizeImage_947523_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:17 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:17 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:17 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c843e496c468799f4b0795d4d6399bf1 with result from task 310
parsl.dataflow.dflow DEBUG: Task 310 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 311 has memoization hash ccafd9e2efc59a520427bff88ffa7f1a
parsl.dataflow.memoization INFO: Task 311 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 1103
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a311ae4c-a558-4bb3-8933-1d97f826ffce_calibrate_947418_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a311ae4c-a558-4bb3-8933-1d97f826ffce_calibrate_947418_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6d4806ce-a683-4104-95c7-cf32e3caafee_characterizeImage_947418_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6d4806ce-a683-4104-95c7-cf32e3caafee_characterizeImage_947418_3.stderr
parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1dd6abe3d1f3dac5c2a6e1c43af920ae with result from task 538
parsl.dataflow.dflow DEBUG: Task 538 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 4562c28f40cc3ee1b2bfef53c922c97e
parsl.dataflow.memoization INFO: Task 539 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1104
parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/702dc96a-2189-49ec-be62-6d573f82ebff_calibrate_947526_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/702dc96a-2189-49ec-be62-6d573f82ebff_calibrate_947526_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/49d1dbc7-dc24-4fcd-a0f8-c7f1ef04c28a_characterizeImage_947526_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/49d1dbc7-dc24-4fcd-a0f8-c7f1ef04c28a_characterizeImage_947526_50.stderr
parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b0c0e0ff62918856ea456698b4cd6ab with result from task 455
parsl.dataflow.dflow DEBUG: Task 455 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f54d2f6-74ff-4640-b343-cef309f18b16_calibrate_947523_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f54d2f6-74ff-4640-b343-cef309f18b16_calibrate_947523_27.stderr
parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc91776784886ecbe5e2bc118a3edcda with result from task 380
parsl.dataflow.dflow DEBUG: Task 380 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d88907cd-299b-4fc3-9231-8d8e09865b98_calibrate_947418_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d88907cd-299b-4fc3-9231-8d8e09865b98_calibrate_947418_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:17 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f8c34c43fbe848abbaeb04d0081e600 with result from task 1132
parsl.dataflow.dflow DEBUG: Task 1132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1133 has memoization hash bc564eebea3c900ce072df3912c4589f
parsl.dataflow.memoization INFO: Task 1133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1133 try 0 launched on executor multi with executor id 1105
parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a8bf300e-b4bb-4c23-b0bf-06da135c9b5a_calibrate_947418_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a8bf300e-b4bb-4c23-b0bf-06da135c9b5a_calibrate_947418_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9ae03857-727f-4791-9d71-5d75367d8f05_characterizeImage_947418_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/9ae03857-727f-4791-9d71-5d75367d8f05_characterizeImage_947418_6.stderr
parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73e955dbea052a72dc20580b6d70a9bd with result from task 736
parsl.dataflow.dflow DEBUG: Task 736 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 3b9fe905ac5969b22008e893b8c2ef95
parsl.dataflow.memoization INFO: Task 737 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 1106
parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/fbf02ae6-6d9a-489f-86f1-0e69fa739582_calibrate_947528_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/fbf02ae6-6d9a-489f-86f1-0e69fa739582_calibrate_947528_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f60f078e-5783-478e-b7d6-49a39d04c10e_characterizeImage_947528_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f60f078e-5783-478e-b7d6-49a39d04c10e_characterizeImage_947528_27.stderr
parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c390f7e3309306653bf2784b435ed92 with result from task 350
parsl.dataflow.dflow DEBUG: Task 350 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5dc8e937-6142-4ff5-a460-1d512288f429_calibrate_947418_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5dc8e937-6142-4ff5-a460-1d512288f429_calibrate_947418_15.stderr
parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22390e0e26e01984fc4d9bc7489a0278 with result from task 245
parsl.dataflow.dflow DEBUG: Task 245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5b7ed98a-fc65-4057-b963-e6689d20c24f_calibrate_947422_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5b7ed98a-fc65-4057-b963-e6689d20c24f_calibrate_947422_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:18 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a15631a6721063be1e9b4414c1573e22 with result from task 1156
parsl.dataflow.dflow DEBUG: Task 1156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1157 has memoization hash 1058ab0b9ee7abab1aeb84ed8f69e51e
parsl.dataflow.memoization INFO: Task 1157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1157 try 0 launched on executor multi with executor id 1107
parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/9fd270f8-fcc2-4d01-8a4c-e6a7e6b87bb8_calibrate_947526_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/9fd270f8-fcc2-4d01-8a4c-e6a7e6b87bb8_calibrate_947526_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/943e68cf-f2d9-4195-bd65-3eefd3d15458_characterizeImage_947526_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/943e68cf-f2d9-4195-bd65-3eefd3d15458_characterizeImage_947526_36.stderr
parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e47dbc82ec06aad4d1e7caa248ab894f with result from task 424
parsl.dataflow.dflow DEBUG: Task 424 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 80654a7031c2d64322bb1d6c0956d313
parsl.dataflow.memoization INFO: Task 425 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f522a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 1108
parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/da32157e-4d58-4496-91ce-9bd627546ab8_calibrate_947422_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/da32157e-4d58-4496-91ce-9bd627546ab8_calibrate_947422_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/02365590-da4e-44ac-8d8d-48404ac57a09_characterizeImage_947422_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/02365590-da4e-44ac-8d8d-48404ac57a09_characterizeImage_947422_45.stderr
parsl.dataflow.dflow INFO: Task 1168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f5ca096af1f690984cfab3cd0bd50e3 with result from task 1168
parsl.dataflow.dflow DEBUG: Task 1168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1169 has memoization hash 8090a3b380c5b5a6714ec4af6137418d
parsl.dataflow.memoization INFO: Task 1169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1169 try 0 launched on executor multi with executor id 1109
parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cde31089-c869-47d9-b075-a6f4b0a3678d_calibrate_947528_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cde31089-c869-47d9-b075-a6f4b0a3678d_calibrate_947528_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d5e82a8-9a73-4c4e-9190-737f8a5e96f0_characterizeImage_947528_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4d5e82a8-9a73-4c4e-9190-737f8a5e96f0_characterizeImage_947528_35.stderr
parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ecf35993b70943de220bab557c9c44c2 with result from task 979
parsl.dataflow.dflow DEBUG: Task 979 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 77323de4c27aaf69795915ad1fda9d93
parsl.dataflow.memoization INFO: Task 980 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 1110
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f826f325-17aa-47c7-a38f-a66226a0fa63_calibrate_947528_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f826f325-17aa-47c7-a38f-a66226a0fa63_calibrate_947528_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6be81840-417b-4eab-9dd6-3c3618534e9f_characterizeImage_947528_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6be81840-417b-4eab-9dd6-3c3618534e9f_characterizeImage_947528_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:18 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 326029bb43d74d70444df6b670f5c4e1 with result from task 1249
parsl.dataflow.dflow DEBUG: Task 1249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1250 has memoization hash 5cc8045e77388a98f4a4edfd0ba2776c
parsl.dataflow.memoization INFO: Task 1250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1250 try 0 launched on executor multi with executor id 1111
parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/9ea592f6-d5d5-441d-bdf0-5891036ece51_calibrate_947418_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/9ea592f6-d5d5-441d-bdf0-5891036ece51_calibrate_947418_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/33df8ce9-a0df-46ca-8fac-5ad26408e8a3_characterizeImage_947418_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/33df8ce9-a0df-46ca-8fac-5ad26408e8a3_characterizeImage_947418_37.stderr
parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa9c0cda603f449e5d83b160d388bf29 with result from task 1078
parsl.dataflow.dflow DEBUG: Task 1078 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 73cee65736590aa768ee97aa13cdba5f
parsl.dataflow.memoization INFO: Task 1079 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1112
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0da68745-6865-4ebe-82bc-2098850d16e8_calibrate_947528_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0da68745-6865-4ebe-82bc-2098850d16e8_calibrate_947528_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af9584ac-a574-4b62-b5b5-9385e381746d_characterizeImage_947528_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/af9584ac-a574-4b62-b5b5-9385e381746d_characterizeImage_947528_12.stderr
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8b3cdf0fe6043f2197ffdf973f54ec6 with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash dc9e557174a63a8e064e41f92bec8768
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 1113
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e7036893-6303-484c-9d4a-c8255ac10c60_calibrate_947524_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e7036893-6303-484c-9d4a-c8255ac10c60_calibrate_947524_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/15d0aefb-d75a-49b4-a810-8fcf2897b6f2_characterizeImage_947524_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/15d0aefb-d75a-49b4-a810-8fcf2897b6f2_characterizeImage_947524_41.stderr
parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59f536a0ba90775ff70eaaa723bd53f4 with result from task 895
parsl.dataflow.dflow DEBUG: Task 895 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 896 has memoization hash f602c3aab909837dd0e8ad4699279b90
parsl.dataflow.memoization INFO: Task 896 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 1114
parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b7196490-e793-4b90-87fa-067f1bc08b3b_calibrate_947528_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b7196490-e793-4b90-87fa-067f1bc08b3b_calibrate_947528_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7a64a0e4-cc08-4cf3-af93-03378a54bde4_characterizeImage_947528_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/7a64a0e4-cc08-4cf3-af93-03378a54bde4_characterizeImage_947528_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:19 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:19 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:19 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 448 try 1 failed
parsl.dataflow.dflow ERROR: Task 448 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca186ada87c983c95b26e9453509ccf with result from task 448
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 449 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 449 try 0 failed
parsl.dataflow.dflow INFO: Task 449 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/75b47438-e23c-4a1a-9a56-a36afe31382b_calibrate_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/75b47438-e23c-4a1a-9a56-a36afe31382b_calibrate_947422_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/5bd66f8a-91e7-4265-9d00-3eb209b7a1bb_characterizeImage_947422_61.stderr
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11bd9bd31c12c4a38914c7028fa494b0 with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 0445280edf8a3af5564d69941e51adc0
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 1115
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c59a2f63-d186-4305-92d5-97bfb35e238b_calibrate_947528_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c59a2f63-d186-4305-92d5-97bfb35e238b_calibrate_947528_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/15046796-af5e-4c6c-b325-e42498e77b7d_characterizeImage_947528_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/15046796-af5e-4c6c-b325-e42498e77b7d_characterizeImage_947528_10.stderr
parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95722257aa7a54edb770810d5483800d with result from task 688
parsl.dataflow.dflow DEBUG: Task 688 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 689 has memoization hash c3503e1bc6620d0cc872fe958c2fa5c8
parsl.dataflow.memoization INFO: Task 689 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 1116
parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e452d87e-b9fb-4699-b4e1-c328f8f8ffdd_calibrate_947526_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e452d87e-b9fb-4699-b4e1-c328f8f8ffdd_calibrate_947526_31.stderr
parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3ac5ecd6-1c7d-4ba1-b4aa-e0d97d91de9c_characterizeImage_947526_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3ac5ecd6-1c7d-4ba1-b4aa-e0d97d91de9c_characterizeImage_947526_31.stderr
parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 726ad2d1579fab5e12cdb3078c40e0c4 with result from task 1081
parsl.dataflow.dflow DEBUG: Task 1081 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 6a533a378e2037bdf89b432cda5740cd
parsl.dataflow.memoization INFO: Task 1082 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 1117
parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/210c0bcd-1ad5-4751-b450-10006654d114_calibrate_947524_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/210c0bcd-1ad5-4751-b450-10006654d114_calibrate_947524_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9e09a2cc-96e3-4ccd-8362-00eda259d74a_characterizeImage_947524_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9e09a2cc-96e3-4ccd-8362-00eda259d74a_characterizeImage_947524_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:20 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3de70864477ae643f53373f248ce047a with result from task 559
parsl.dataflow.dflow DEBUG: Task 559 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 462b0d7d6ce20e7635d7e9205b3cb8f8
parsl.dataflow.memoization INFO: Task 560 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 1118
parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5a063e76-ed01-4a14-802a-80f83a3c0ca2_calibrate_947528_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5a063e76-ed01-4a14-802a-80f83a3c0ca2_calibrate_947528_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/91b0cb1d-2da7-4a38-babc-cbcfa1ba9a0d_characterizeImage_947528_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/91b0cb1d-2da7-4a38-babc-cbcfa1ba9a0d_characterizeImage_947528_22.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7576c0faf337d35a7c231a1813cc46f1 with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/772b2184-dfe5-45c4-acb9-2d2b3980310a_calibrate_947524_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/772b2184-dfe5-45c4-acb9-2d2b3980310a_calibrate_947524_23.stderr
parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f3d540157dc72a34a79caeed9d45671 with result from task 271
parsl.dataflow.dflow DEBUG: Task 271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 20509b79b605ffad5bb0f3173eb06e6d
parsl.dataflow.memoization INFO: Task 272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 1119
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6cc014df-c884-477a-9564-47e59fd0bab8_calibrate_947418_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6cc014df-c884-477a-9564-47e59fd0bab8_calibrate_947418_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d9e00eb6-219b-4bd3-9523-c574f8ed66ec_characterizeImage_947418_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d9e00eb6-219b-4bd3-9523-c574f8ed66ec_characterizeImage_947418_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:20 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 224e52a84abdbf5688956a6437c648f8 with result from task 649
parsl.dataflow.dflow DEBUG: Task 649 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 6c4b2e796a9a84db0f7faf2569583513
parsl.dataflow.memoization INFO: Task 650 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e34fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 1120
parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/76aae12f-d989-4ea0-9dd6-77126031994d_calibrate_947422_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/76aae12f-d989-4ea0-9dd6-77126031994d_calibrate_947422_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d8cb596b-f33d-482a-97f2-28c8f379292f_characterizeImage_947422_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d8cb596b-f33d-482a-97f2-28c8f379292f_characterizeImage_947422_16.stderr
parsl.dataflow.dflow INFO: Task 1216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c623794dd617f419b410841660631efb with result from task 1216
parsl.dataflow.dflow DEBUG: Task 1216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1217 has memoization hash 88053e20539613fe002a95148439b646
parsl.dataflow.memoization INFO: Task 1217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1217 try 0 launched on executor multi with executor id 1121
parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79f375ca-4805-43f9-a5b1-95e9a74bb56c_calibrate_947418_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79f375ca-4805-43f9-a5b1-95e9a74bb56c_calibrate_947418_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/80a41f13-e67c-4514-b3c1-c009d06bfb2d_characterizeImage_947418_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/80a41f13-e67c-4514-b3c1-c009d06bfb2d_characterizeImage_947418_26.stderr
parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d6dc6a0a62481388f1b8b5e71e63820 with result from task 716
parsl.dataflow.dflow DEBUG: Task 716 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0dda74e6-b3a9-4f6c-ac86-c3131c013fcc_calibrate_947523_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0dda74e6-b3a9-4f6c-ac86-c3131c013fcc_calibrate_947523_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:21 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7d81e722af09c6dfd397040ac216cc7 with result from task 1021
parsl.dataflow.dflow DEBUG: Task 1021 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 59978bc752e0b9581445b4c4fae9e387
parsl.dataflow.memoization INFO: Task 1022 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1122
parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7346fe4d-dc7d-436d-95b6-e1b30eb45bb9_calibrate_947418_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7346fe4d-dc7d-436d-95b6-e1b30eb45bb9_calibrate_947418_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6a6cf7f1-4b2d-4567-8941-27ab00c568a0_characterizeImage_947418_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/6a6cf7f1-4b2d-4567-8941-27ab00c568a0_characterizeImage_947418_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:21 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e75c8d717664da82441a13ad0d1c3be4 with result from task 1084
parsl.dataflow.dflow DEBUG: Task 1084 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash eef86ce413aaa29579dc4bd0af106e28
parsl.dataflow.memoization INFO: Task 1085 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1123
parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c0196206-22c3-477f-b03b-ba5686168b0e_calibrate_947528_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c0196206-22c3-477f-b03b-ba5686168b0e_calibrate_947528_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/37eef965-1ef3-47dd-b485-1f82383d4ed0_characterizeImage_947528_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/37eef965-1ef3-47dd-b485-1f82383d4ed0_characterizeImage_947528_24.stderr
parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d161ec43bad3e0845b4f939268d566d8 with result from task 722
parsl.dataflow.dflow DEBUG: Task 722 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/53aacb15-efed-44d4-8caf-6a7664f49a55_calibrate_947422_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/53aacb15-efed-44d4-8caf-6a7664f49a55_calibrate_947422_48.stderr
parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42de8e80a901f4edb75c8425b003fad8 with result from task 836
parsl.dataflow.dflow DEBUG: Task 836 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1c60deba-0570-4fde-88d1-1f78c4f8166d_calibrate_947418_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1c60deba-0570-4fde-88d1-1f78c4f8166d_calibrate_947418_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:21 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 154e9bb4206d21a6c442242869bf5959 with result from task 547
parsl.dataflow.dflow DEBUG: Task 547 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 2396a2412423cd42124eca4a9ab584fa
parsl.dataflow.memoization INFO: Task 548 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 1124
parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b05941a3-5b7f-4309-8cc3-973524852788_calibrate_947526_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b05941a3-5b7f-4309-8cc3-973524852788_calibrate_947526_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b490c1e5-afec-457d-bcc8-b88142df2778_characterizeImage_947526_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b490c1e5-afec-457d-bcc8-b88142df2778_characterizeImage_947526_47.stderr
parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 143bc622b157e625b4c0cb978c8c6be7 with result from task 521
parsl.dataflow.dflow DEBUG: Task 521 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ba6404b9-5b95-4a6d-9578-b94d264c873f_calibrate_947523_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ba6404b9-5b95-4a6d-9578-b94d264c873f_calibrate_947523_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:22 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 701668ddbc648bb8276e61c9b78a1c58 with result from task 791
parsl.dataflow.dflow DEBUG: Task 791 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/88c6a893-693c-47d8-800f-487dcf993011_calibrate_947422_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/88c6a893-693c-47d8-800f-487dcf993011_calibrate_947422_24.stderr
parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0038ae9c9eaca57c1376dfbcff45c663 with result from task 668
parsl.dataflow.dflow DEBUG: Task 668 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5c82cec6-4658-4943-ac9b-f9ea8eb03b61_calibrate_947420_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5c82cec6-4658-4943-ac9b-f9ea8eb03b61_calibrate_947420_28.stderr
parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c022b5ed9b43da6794605ff3887fe102 with result from task 709
parsl.dataflow.dflow DEBUG: Task 709 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 710 has memoization hash d63d85589370d9bd0050b0c9871eff61
parsl.dataflow.memoization INFO: Task 710 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 1125
parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/958d7d95-7b15-4126-8ff7-6403b7e115d6_calibrate_947526_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/958d7d95-7b15-4126-8ff7-6403b7e115d6_calibrate_947526_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ae01a9ff-e074-4369-9f52-48ecd84bfda2_characterizeImage_947526_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ae01a9ff-e074-4369-9f52-48ecd84bfda2_characterizeImage_947526_5.stderr
parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 875086ce3fecc977d38147635d9a9bda with result from task 656
parsl.dataflow.dflow DEBUG: Task 656 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/621581a6-6eed-4ab8-89dc-e9567f5134ec_calibrate_947523_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/621581a6-6eed-4ab8-89dc-e9567f5134ec_calibrate_947523_4.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a006a6f62e80c5636837fe5bef55948f with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/385130ac-5b5a-431e-a51a-8f305db698f0_calibrate_947422_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/385130ac-5b5a-431e-a51a-8f305db698f0_calibrate_947422_33.stderr
parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b50a48ae4c7e896f9be9ca8b9882cca with result from task 793
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:22 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 793 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 50e8839ce009417778d1ab892bc7d676
parsl.dataflow.memoization INFO: Task 794 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1126
parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/549c0d14-eefa-40b9-8100-a1abe20a15a7_calibrate_947422_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/549c0d14-eefa-40b9-8100-a1abe20a15a7_calibrate_947422_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/07014230-0fb5-431f-a72b-09a357cea158_characterizeImage_947422_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/07014230-0fb5-431f-a72b-09a357cea158_characterizeImage_947422_19.stderr
parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd8b0df220b09d0a8eb846d0213f400e with result from task 704
parsl.dataflow.dflow DEBUG: Task 704 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5a3325fa-dff2-42bb-9d8f-b66e80d1c459_calibrate_947420_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5a3325fa-dff2-42bb-9d8f-b66e80d1c459_calibrate_947420_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:23 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b5a133677ece71e360976e70a85e0e4 with result from task 467
parsl.dataflow.dflow DEBUG: Task 467 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f697da3-d6be-43f1-aea5-1318f3b201b4_calibrate_947523_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f697da3-d6be-43f1-aea5-1318f3b201b4_calibrate_947523_45.stderr
parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1283dc738ac86b881f94a11cc0eb304b with result from task 785
parsl.dataflow.dflow DEBUG: Task 785 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/8fe99e24-71e0-4cf7-a51d-4a90d5fc51f9_calibrate_947523_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/8fe99e24-71e0-4cf7-a51d-4a90d5fc51f9_calibrate_947523_49.stderr
parsl.dataflow.dflow INFO: Task 1123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db84556e20fcd3d40b6ee53d048521ae with result from task 1123
parsl.dataflow.dflow DEBUG: Task 1123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1124 has memoization hash 782a67ebeb0f11a2fc2f6d2c909ee1c0
parsl.dataflow.memoization INFO: Task 1124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1124 try 0 launched on executor multi with executor id 1127
parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f0182558-4b38-4ec4-a387-8173b1a68395_calibrate_947418_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f0182558-4b38-4ec4-a387-8173b1a68395_calibrate_947418_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/35eeb198-2d1a-4a9b-b570-082ed8ed2179_characterizeImage_947418_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/35eeb198-2d1a-4a9b-b570-082ed8ed2179_characterizeImage_947418_23.stderr
parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5824fc21038906a4f1da458cf963339f with result from task 581
parsl.dataflow.dflow DEBUG: Task 581 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9975fec-0fbe-47aa-93f2-fcd9fd419228_calibrate_947420_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9975fec-0fbe-47aa-93f2-fcd9fd419228_calibrate_947420_55.stderr
parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 900403fc1a81c391f790b1606dcdfdf0 with result from task 376
parsl.dataflow.dflow DEBUG: Task 376 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 377 has memoization hash d6ce0cc7422d21eded29b38add7c659b
parsl.dataflow.memoization INFO: Task 377 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f23060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 1128
parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/65b7d837-ca27-4629-988a-fc8d46887d5c_calibrate_947524_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/65b7d837-ca27-4629-988a-fc8d46887d5c_calibrate_947524_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b4c64ea9-78e2-4331-a008-ab2d7a99ea7c_characterizeImage_947524_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b4c64ea9-78e2-4331-a008-ab2d7a99ea7c_characterizeImage_947524_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:23 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90e9f23a19ed26d4e28ba673e02a6f44 with result from task 766
parsl.dataflow.dflow DEBUG: Task 766 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 088bf45f8d4cc4bcf3f5b988b3010180
parsl.dataflow.memoization INFO: Task 767 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 1129
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5cf5ae4e-36f3-4897-aee7-1db344dc269e_calibrate_947528_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5cf5ae4e-36f3-4897-aee7-1db344dc269e_calibrate_947528_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/abad35b8-a904-436a-bd7f-a27a4fa90057_characterizeImage_947528_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/abad35b8-a904-436a-bd7f-a27a4fa90057_characterizeImage_947528_14.stderr
parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2801210b8dacdd2301420761917826a3 with result from task 829
parsl.dataflow.dflow DEBUG: Task 829 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 830 has memoization hash ae3d38de4e3fee33280a9cd7a27ce728
parsl.dataflow.memoization INFO: Task 830 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 1130
parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/7a1ebbbb-8d12-465f-94e7-b66ff1a516f7_calibrate_947420_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/7a1ebbbb-8d12-465f-94e7-b66ff1a516f7_calibrate_947420_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/3bcf33d6-a23c-4c32-9eb2-bfd4f16bb310_characterizeImage_947420_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/3bcf33d6-a23c-4c32-9eb2-bfd4f16bb310_characterizeImage_947420_38.stderr
parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c65c0b887f9c76c85cbff1e631870faf with result from task 911
parsl.dataflow.dflow DEBUG: Task 911 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/32dd0021-2ea0-49e8-9818-9cd56f3ad929_calibrate_947422_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/32dd0021-2ea0-49e8-9818-9cd56f3ad929_calibrate_947422_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:23 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 113633036f1d48bdc34ae33e2b8fc67f with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/956e513c-2a54-4391-8f2e-ce5a4af632be_calibrate_947523_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/956e513c-2a54-4391-8f2e-ce5a4af632be_calibrate_947523_21.stderr
parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be3c4348063610ffabdad6fdf54dfe8d with result from task 1096
parsl.dataflow.dflow DEBUG: Task 1096 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash a721d2640ac47ade5644b6d22f729ad6
parsl.dataflow.memoization INFO: Task 1097 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 1131
parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a891c0df-30ef-4902-b618-34bf467e87bf_calibrate_947524_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a891c0df-30ef-4902-b618-34bf467e87bf_calibrate_947524_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fb3fb52-e7a0-41f4-bba1-6405f49923af_characterizeImage_947524_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fb3fb52-e7a0-41f4-bba1-6405f49923af_characterizeImage_947524_12.stderr
parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6bedbd974709293ad328bc5fc5ae201 with result from task 701
parsl.dataflow.dflow DEBUG: Task 701 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5038707c-3058-4a1f-8cb7-d4d6cfbd2647_calibrate_947420_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5038707c-3058-4a1f-8cb7-d4d6cfbd2647_calibrate_947420_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:24 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4aa61aa71ec93e4e580809be7a38f5f with result from task 956
parsl.dataflow.dflow DEBUG: Task 956 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8e5b8ab7-5f54-4bfc-89c6-c5b348caddc8_calibrate_947420_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/8e5b8ab7-5f54-4bfc-89c6-c5b348caddc8_calibrate_947420_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:24 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4e9431a629039b7b30097f391f9297b with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6762cca3-aa1f-49bb-b2ca-f190238663d3_calibrate_947418_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6762cca3-aa1f-49bb-b2ca-f190238663d3_calibrate_947418_14.stderr
parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e529b150ba626372a73492955867ec8f with result from task 413
parsl.dataflow.dflow DEBUG: Task 413 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1c76ea0c-6a27-49da-85b7-4ce9c52985c8_calibrate_947523_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1c76ea0c-6a27-49da-85b7-4ce9c52985c8_calibrate_947523_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:24 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5a67aafe485aba39dcba33172c63e252 with result from task 1108
parsl.dataflow.dflow DEBUG: Task 1108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 10ed70bb29431029b3871b780ba832ec
parsl.dataflow.memoization INFO: Task 1109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1132
parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c7ab8636-3ec7-400a-a51e-31d1a2b3f6d4_calibrate_947418_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c7ab8636-3ec7-400a-a51e-31d1a2b3f6d4_calibrate_947418_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a203b6f9-fbbe-45f5-ab53-24c1b5f3cf8c_characterizeImage_947418_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/a203b6f9-fbbe-45f5-ab53-24c1b5f3cf8c_characterizeImage_947418_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:25 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3ce8f2716401a048a0d82ee72e56267 with result from task 197
parsl.dataflow.dflow DEBUG: Task 197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9293aa8a-2656-4501-a2a9-bed79196edec_calibrate_947420_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9293aa8a-2656-4501-a2a9-bed79196edec_calibrate_947420_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:25 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2fb55bd3082c8214c12a646b84489e9 with result from task 1271
parsl.dataflow.dflow DEBUG: Task 1271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/68720f84-0af8-4548-a790-96772b0f3554_calibrate_947422_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/68720f84-0af8-4548-a790-96772b0f3554_calibrate_947422_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:26 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6ad410cd0db13250d17d9deaa71efe5 with result from task 470
parsl.dataflow.dflow DEBUG: Task 470 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/777795b7-4dda-4daf-8c88-cc36bfc23f84_calibrate_947420_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/777795b7-4dda-4daf-8c88-cc36bfc23f84_calibrate_947420_26.stderr
parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3eba08dbbd386297da396b431648d7 with result from task 1030
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:26 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 1030 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash 6ac99585b8e65d182271359ff83f7771
parsl.dataflow.memoization INFO: Task 1031 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 1133
parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cccdd7b6-19b1-4563-8bb6-2a6b0ec82fe9_calibrate_947528_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cccdd7b6-19b1-4563-8bb6-2a6b0ec82fe9_calibrate_947528_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a02621e1-b4a6-4216-95c6-fe311a04c979_characterizeImage_947528_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a02621e1-b4a6-4216-95c6-fe311a04c979_characterizeImage_947528_13.stderr
parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 39692de7e0c8dd1e29e593e5cf0fc604 with result from task 206
parsl.dataflow.dflow DEBUG: Task 206 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/dac0b60a-43bb-4aa8-aa60-34415d2730de_calibrate_947523_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/dac0b60a-43bb-4aa8-aa60-34415d2730de_calibrate_947523_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:26 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa4f3655db55378b3c4e9c93819b2d73 with result from task 797
parsl.dataflow.dflow DEBUG: Task 797 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e464522-6c1d-4c8e-9499-be303508e0a7_calibrate_947420_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/2e464522-6c1d-4c8e-9499-be303508e0a7_calibrate_947420_14.stderr
parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 345feffb4c6f85ad349893bc29d93d8a with result from task 949
parsl.dataflow.dflow DEBUG: Task 949 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 950 has memoization hash f25881d8d2d424825d3b48f2bd1773fb
parsl.dataflow.memoization INFO: Task 950 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 1134
parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4548627b-a598-4bb8-9752-7301fa381524_calibrate_947420_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4548627b-a598-4bb8-9752-7301fa381524_calibrate_947420_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/da32664d-e39c-4608-af92-02fdbedcf32c_characterizeImage_947420_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/da32664d-e39c-4608-af92-02fdbedcf32c_characterizeImage_947420_47.stderr
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccc1b90db63911f6a3d57be861aeaae7 with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 7724eb64d16d4d8d43cf87e6c2f1e6e8
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 1135
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a4eefb8b-c9c9-44d4-9327-7fb95bf42f04_calibrate_947528_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a4eefb8b-c9c9-44d4-9327-7fb95bf42f04_calibrate_947528_47.stderr
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4f448ad2-ff64-4f0d-a9fe-8d35c3717e60_characterizeImage_947528_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/4f448ad2-ff64-4f0d-a9fe-8d35c3717e60_characterizeImage_947528_47.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5d303141e6f6c3f6402b0b2f50ad21f with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/917fc988-75b6-4437-9356-b2d8d56fbfb2_calibrate_947418_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/917fc988-75b6-4437-9356-b2d8d56fbfb2_calibrate_947418_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:27 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 689a929ba92da761d7d851103fafa457 with result from task 773
parsl.dataflow.dflow DEBUG: Task 773 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/835b39ff-5ab5-48c0-8e2e-d6b8b64a8d32_calibrate_947420_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/835b39ff-5ab5-48c0-8e2e-d6b8b64a8d32_calibrate_947420_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:27 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:27 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51efdc48f176285545c36605f12cc5d3 with result from task 749
parsl.dataflow.dflow DEBUG: Task 749 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5e14787e-45bc-440c-aa74-e66129e82c9c_calibrate_947523_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5e14787e-45bc-440c-aa74-e66129e82c9c_calibrate_947523_38.stderr
parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d5147f8da71c9325c4b9c366ad63142 with result from task 403
parsl.dataflow.dflow DEBUG: Task 403 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 0760abbbe6f858111d0d6e04307cad2e
parsl.dataflow.memoization INFO: Task 404 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f50cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 1136
parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b0b4355-425f-4011-b705-788bf9247086_calibrate_947523_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b0b4355-425f-4011-b705-788bf9247086_calibrate_947523_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8a0ef0ed-c4f5-49d9-8065-653d13cc1b2b_characterizeImage_947523_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8a0ef0ed-c4f5-49d9-8065-653d13cc1b2b_characterizeImage_947523_11.stderr
parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc70190ab755949f712ab340224ee97c with result from task 494
parsl.dataflow.dflow DEBUG: Task 494 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c77ff5cc-70d2-47f8-9db8-abcdfa782dd2_calibrate_947418_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c77ff5cc-70d2-47f8-9db8-abcdfa782dd2_calibrate_947418_34.stderr
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2bafb49296a8ddf3dd3faf2fdce2be8a with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9202426-db5d-49ef-b961-037ed86f567f_calibrate_947420_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d9202426-db5d-49ef-b961-037ed86f567f_calibrate_947420_7.stderr
parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 812d18db21f2f8732847befbc29b3082 with result from task 859
parsl.dataflow.dflow DEBUG: Task 859 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 0732022401410620dc72e4f27c60f872
parsl.dataflow.memoization INFO: Task 860 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 1137
parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/736eb83a-b81a-4a2d-b86d-28dc6933b0cd_calibrate_947422_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/736eb83a-b81a-4a2d-b86d-28dc6933b0cd_calibrate_947422_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/398e9a70-353e-4379-8654-b8dcd5a3e6c5_characterizeImage_947422_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/398e9a70-353e-4379-8654-b8dcd5a3e6c5_characterizeImage_947422_51.stderr
parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60cb6892fee933664043040d90f45ce4 with result from task 638
parsl.dataflow.dflow DEBUG: Task 638 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f20df706-294f-4b1b-b2a8-f8d6c86ce07e_calibrate_947420_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f20df706-294f-4b1b-b2a8-f8d6c86ce07e_calibrate_947420_41.stderr
parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b147cf768fb831e25dd7897d538763c with result from task 1060
parsl.dataflow.dflow DEBUG: Task 1060 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 70275317fac18b96b63f5d1bbaa173b2
parsl.dataflow.memoization INFO: Task 1061 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1138
parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/431e9228-2c4b-438d-aa25-ddb70972d061_calibrate_947526_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/431e9228-2c4b-438d-aa25-ddb70972d061_calibrate_947526_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fc1fa03b-6b24-4369-a628-9e99a47ab8cb_characterizeImage_947526_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fc1fa03b-6b24-4369-a628-9e99a47ab8cb_characterizeImage_947526_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 352
parsl.dataflow.strategy DEBUG: Executor multi has 352 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:28:28 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c30cd1bdb7e1fe5460908f7b28039ee with result from task 401
parsl.dataflow.dflow DEBUG: Task 401 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a908ce4-a357-4bd0-bdd6-5e1f76d02f58_calibrate_947523_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a908ce4-a357-4bd0-bdd6-5e1f76d02f58_calibrate_947523_8.stderr
2024-03-14 07:28:28 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 241335ebd86fab942aae5782f2177628 with result from task 437
parsl.dataflow.dflow DEBUG: Task 437 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/29deaaaa-2eee-48d9-8ecd-331598e38e59_calibrate_947524_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/29deaaaa-2eee-48d9-8ecd-331598e38e59_calibrate_947524_15.stderr
parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8590eb7b1dd6348e8bc5c5ce6c40c8da with result from task 338
parsl.dataflow.dflow DEBUG: Task 338 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dbb0c4e7-7d1c-42f7-8d00-5b93a7e6454f_calibrate_947418_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dbb0c4e7-7d1c-42f7-8d00-5b93a7e6454f_calibrate_947418_58.stderr
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afd5f3c3ec3e83324c73cbaef6d7117b with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/831e3521-2152-483d-b80e-cea8639ea31c_calibrate_947524_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/831e3521-2152-483d-b80e-cea8639ea31c_calibrate_947524_36.stderr
parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62d80ac1480c9d7bbbf18b3ec6ecdd8b with result from task 523
parsl.dataflow.dflow DEBUG: Task 523 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 524 has memoization hash aae27c829ba7bd7a1ed7e11cd676922d
parsl.dataflow.memoization INFO: Task 524 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 1139
parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/4f9f8070-b52e-436c-95e3-4f7b515cd9f0_calibrate_947422_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/4f9f8070-b52e-436c-95e3-4f7b515cd9f0_calibrate_947422_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/98aef122-daa5-4bf0-a3d4-0fc33681dd7c_characterizeImage_947422_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/98aef122-daa5-4bf0-a3d4-0fc33681dd7c_characterizeImage_947422_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:29 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7315e77a2a4947307e61bf2053bdf2a with result from task 652
parsl.dataflow.dflow DEBUG: Task 652 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 653 has memoization hash cd6f1d5b2c1c3e6b2870b3892028852e
parsl.dataflow.memoization INFO: Task 653 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 1140
parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/09669497-a231-46b0-9c97-f238ad76aacd_calibrate_947526_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/09669497-a231-46b0-9c97-f238ad76aacd_calibrate_947526_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f2f62467-87c0-48a9-864c-0e04a4aa5b2c_characterizeImage_947526_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f2f62467-87c0-48a9-864c-0e04a4aa5b2c_characterizeImage_947526_26.stderr
parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19466c7321a03ec50517cd21585f78ab with result from task 853
parsl.dataflow.dflow DEBUG: Task 853 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 84b4befb410f729cf08072fb11603a96
parsl.dataflow.memoization INFO: Task 854 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 1141
parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/562cd549-a53a-4d1d-bd2b-73d9c199cba7_calibrate_947524_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/562cd549-a53a-4d1d-bd2b-73d9c199cba7_calibrate_947524_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4ec997f2-ef1e-42ab-840a-f3106bfcde70_characterizeImage_947524_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4ec997f2-ef1e-42ab-840a-f3106bfcde70_characterizeImage_947524_51.stderr
parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67abe870c46ea006d232c963128f1290 with result from task 1033
parsl.dataflow.dflow DEBUG: Task 1033 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 7ddeb76f042b8119639e1305dee0f798
parsl.dataflow.memoization INFO: Task 1034 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 1142
parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/63920ee9-68e7-4132-ac1a-4eef95756baa_calibrate_947422_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/63920ee9-68e7-4132-ac1a-4eef95756baa_calibrate_947422_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/c23b2dfd-514c-4888-a1b4-bd5a894ca416_characterizeImage_947422_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/c23b2dfd-514c-4888-a1b4-bd5a894ca416_characterizeImage_947422_50.stderr
parsl.dataflow.dflow INFO: Task 1139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e47c85894fc05fe475fd8d81ac1c87a with result from task 1139
parsl.dataflow.dflow DEBUG: Task 1139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e58ea7d9-01f5-462b-bb92-1b9fb73c1bde_calibrate_947418_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e58ea7d9-01f5-462b-bb92-1b9fb73c1bde_calibrate_947418_36.stderr
parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7452aa73e875d990a9088c5b6e5c2c1 with result from task 872
parsl.dataflow.dflow DEBUG: Task 872 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/79a2bc45-b4bf-4674-bf7d-b1b6ac5ae337_calibrate_947420_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/79a2bc45-b4bf-4674-bf7d-b1b6ac5ae337_calibrate_947420_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:29 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cd69b109d7ddbc78aeface14b35804a with result from task 452
parsl.dataflow.dflow DEBUG: Task 452 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9a2be125-367a-4b33-9b4c-fdfbb1fb0812_calibrate_947420_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9a2be125-367a-4b33-9b4c-fdfbb1fb0812_calibrate_947420_48.stderr
parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bdf72acb9fd71f5d729bd99dd5dda967 with result from task 619
parsl.dataflow.dflow DEBUG: Task 619 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 1baee34f9b75c7be236cf38d515fb28b
parsl.dataflow.memoization INFO: Task 620 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 1143
parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0792b4e1-2dc1-45e1-ab04-f36921d2c00f_calibrate_947526_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0792b4e1-2dc1-45e1-ab04-f36921d2c00f_calibrate_947526_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/774ee501-80d2-4a26-a178-588780a94fad_characterizeImage_947526_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/774ee501-80d2-4a26-a178-588780a94fad_characterizeImage_947526_60.stderr
parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0729ff4952a5b56e71da3188fac0508 with result from task 371
parsl.dataflow.dflow DEBUG: Task 371 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d0c4c75e-16c5-46e5-bfe2-d4b103f9fe19_calibrate_947420_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d0c4c75e-16c5-46e5-bfe2-d4b103f9fe19_calibrate_947420_53.stderr
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 370264e1c86d44e8cad5e4cd5af89c79 with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 0bbb1893f4a6916f67f260d8dfe43549
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 1144
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a6138eac-9f4a-4ac4-afcc-a1fcf608d04c_calibrate_947422_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a6138eac-9f4a-4ac4-afcc-a1fcf608d04c_calibrate_947422_27.stderr
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6ef38294-9cbd-4387-b71c-0582e9a6f704_characterizeImage_947422_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/6ef38294-9cbd-4387-b71c-0582e9a6f704_characterizeImage_947422_27.stderr
parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5dd3c7612ff98dfba8ffabdc24cad121 with result from task 706
parsl.dataflow.dflow DEBUG: Task 706 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 707 has memoization hash f06667f51cfd9919696d3d2da15a33bb
parsl.dataflow.memoization INFO: Task 707 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 1145
parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f9f4e8dd-481a-4200-9552-6ade6dfb2e66_calibrate_947422_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f9f4e8dd-481a-4200-9552-6ade6dfb2e66_calibrate_947422_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8e5d6d7a-a819-4881-84cb-b1bc4ea11edd_characterizeImage_947422_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8e5d6d7a-a819-4881-84cb-b1bc4ea11edd_characterizeImage_947422_32.stderr
parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0775dd69de2f079c9945ce68d66c365 with result from task 571
parsl.dataflow.dflow DEBUG: Task 571 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 52d16b34deb771599acf23300f1e0927
parsl.dataflow.memoization INFO: Task 572 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 1146
parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ca0a7610-86d3-4891-bd87-a2183f83628d_calibrate_947420_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ca0a7610-86d3-4891-bd87-a2183f83628d_calibrate_947420_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4aef455a-325e-4ac3-aa13-c609ee59a344_characterizeImage_947420_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4aef455a-325e-4ac3-aa13-c609ee59a344_characterizeImage_947420_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:30 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 027f5ba53fd7b19f8a17c57ff9880e63 with result from task 674
parsl.dataflow.dflow DEBUG: Task 674 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e374fdd1-928d-4a47-8c49-a0c0b86b63cf_calibrate_947422_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e374fdd1-928d-4a47-8c49-a0c0b86b63cf_calibrate_947422_29.stderr
parsl.dataflow.dflow INFO: Task 1180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b98d32da36496df39353d8fc01af3576 with result from task 1180
parsl.dataflow.dflow DEBUG: Task 1180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1181 has memoization hash 60847ce4cbe10e0e9e6552102ee9af2b
parsl.dataflow.memoization INFO: Task 1181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1181 try 0 launched on executor multi with executor id 1147
parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/caec9a05-50ab-4127-9ca0-22e1f53bceb0_calibrate_947523_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/caec9a05-50ab-4127-9ca0-22e1f53bceb0_calibrate_947523_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a8e39e93-b5b5-4a1b-ab69-c2d454edafae_characterizeImage_947523_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a8e39e93-b5b5-4a1b-ab69-c2d454edafae_characterizeImage_947523_12.stderr
parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f568fc097a413e22adf3eba3239a3999 with result from task 874
parsl.dataflow.dflow DEBUG: Task 874 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 5464f95d50d8c67c023f0601a9c929b2
parsl.dataflow.memoization INFO: Task 875 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d0bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 1148
parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6ca1c8bc-9e06-4c5f-9072-46243fe01fbc_calibrate_947528_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6ca1c8bc-9e06-4c5f-9072-46243fe01fbc_calibrate_947528_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cc139760-7b9a-427a-b6c6-a722ea5cf2f9_characterizeImage_947528_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cc139760-7b9a-427a-b6c6-a722ea5cf2f9_characterizeImage_947528_11.stderr
parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2be91291622d8c681e678de44778e06 with result from task 937
parsl.dataflow.dflow DEBUG: Task 937 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 938 has memoization hash 7ac32e0e59d9d89078a6616cef4d9473
parsl.dataflow.memoization INFO: Task 938 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1149
parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/196bdee1-4400-4596-a291-ea369e596d87_calibrate_947524_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/196bdee1-4400-4596-a291-ea369e596d87_calibrate_947524_26.stderr
parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a32d3dd-a04c-436c-8da4-d20c59623366_characterizeImage_947524_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a32d3dd-a04c-436c-8da4-d20c59623366_characterizeImage_947524_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:30 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1140f05a98bcbf106825ef9df45b32c4 with result from task 847
parsl.dataflow.dflow DEBUG: Task 847 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 848 has memoization hash e6e81464f824e86d0feaf9aa48870218
parsl.dataflow.memoization INFO: Task 848 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d09ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 1150
parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9e1792f7-3284-4ae2-a382-742170b50fd1_calibrate_947524_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9e1792f7-3284-4ae2-a382-742170b50fd1_calibrate_947524_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8e1f7857-822e-4f41-b8e5-99be0a8902a4_characterizeImage_947524_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/8e1f7857-822e-4f41-b8e5-99be0a8902a4_characterizeImage_947524_4.stderr
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7256ed985466c2b86bb6c8269bb7784 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dfec4333-7203-4eb6-b3fd-4b4461dc2a77_calibrate_947418_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dfec4333-7203-4eb6-b3fd-4b4461dc2a77_calibrate_947418_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:30 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:31 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5e40e3dbd0bcac3f72d6c9e5b30277f with result from task 1100
parsl.dataflow.dflow DEBUG: Task 1100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/07eaadda-3d87-4a5b-96df-93735c2ab836_calibrate_947523_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/07eaadda-3d87-4a5b-96df-93735c2ab836_calibrate_947523_37.stderr
parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 120d471a4d52bfb9ee1384be220b41b5 with result from task 1090
parsl.dataflow.dflow DEBUG: Task 1090 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 564b4ad8c33f40a7601327757fb84814
parsl.dataflow.memoization INFO: Task 1091 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c35d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1151
parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dd845476-aa5d-491c-b7f3-d1105e4561e9_calibrate_947524_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dd845476-aa5d-491c-b7f3-d1105e4561e9_calibrate_947524_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fd4df93-474d-498d-9eba-b56ebc1adeb5_characterizeImage_947524_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7fd4df93-474d-498d-9eba-b56ebc1adeb5_characterizeImage_947524_56.stderr
parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a1b8cd575c1af764417f7697d5cb41d3 with result from task 505
parsl.dataflow.dflow DEBUG: Task 505 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 39fbb77024ba043d27e67537bc9b8620
parsl.dataflow.memoization INFO: Task 506 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 1152
parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/33caab7d-a1fe-4e84-b563-a0021c0ee47f_calibrate_947524_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/33caab7d-a1fe-4e84-b563-a0021c0ee47f_calibrate_947524_13.stderr
parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9cca75ac-c5cc-4208-ac32-9f7b7b6584eb_characterizeImage_947524_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9cca75ac-c5cc-4208-ac32-9f7b7b6584eb_characterizeImage_947524_13.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aaaeb72ca18fff5e20d0543e57a6e1ee with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 3b396b5cd02af66caaef2e7d0287521e
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 1153
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0508b3ba-7a72-4d56-83c2-b9c44ff24a09_calibrate_947422_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0508b3ba-7a72-4d56-83c2-b9c44ff24a09_calibrate_947422_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1c12f9cd-349c-4f5d-9384-f753542ff6f8_characterizeImage_947422_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/1c12f9cd-349c-4f5d-9384-f753542ff6f8_characterizeImage_947422_60.stderr
parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2da7ba8425650b55538223bec47cff with result from task 191
parsl.dataflow.dflow DEBUG: Task 191 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f0f90091-3c67-40a0-86e3-b5db5067408b_calibrate_947523_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f0f90091-3c67-40a0-86e3-b5db5067408b_calibrate_947523_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:31 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 481cbc12298e63faac56f2f1b4c9d55e with result from task 313
parsl.dataflow.dflow DEBUG: Task 313 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 314 has memoization hash b879c10b1d44303bf0ce9c69ed29f6a5
parsl.dataflow.memoization INFO: Task 314 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 1154
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3429d20a-e0a2-49c2-b24a-6ab4d83ee35c_calibrate_947420_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3429d20a-e0a2-49c2-b24a-6ab4d83ee35c_calibrate_947420_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7d8709ba-fde6-4ba4-9c16-a768a61a9f92_characterizeImage_947420_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/7d8709ba-fde6-4ba4-9c16-a768a61a9f92_characterizeImage_947420_56.stderr
parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca2fba3ff01344122fe2feb4ada20e4f with result from task 680
parsl.dataflow.dflow DEBUG: Task 680 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/ff3d3c31-f6f9-49d5-b046-c27e66917e2c_calibrate_947422_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/ff3d3c31-f6f9-49d5-b046-c27e66917e2c_calibrate_947422_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:32 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 08698e97273c58875b187f7347172945 with result from task 826
parsl.dataflow.dflow DEBUG: Task 826 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 7b272cdb0766459285ae61e835457ec9
parsl.dataflow.memoization INFO: Task 827 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d08900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 1155
parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/195c6bb2-b831-451b-b169-3e52d6a56dc2_calibrate_947528_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/195c6bb2-b831-451b-b169-3e52d6a56dc2_calibrate_947528_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/78a60f3b-a900-49eb-958d-43eafffb8900_characterizeImage_947528_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/78a60f3b-a900-49eb-958d-43eafffb8900_characterizeImage_947528_56.stderr
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72c5e640fc9990491cee0c5ee4caf204 with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 730df4dbbcfdd214f72f824e09f2f89f
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 1156
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bbe4e584-c2d3-4b33-a8d5-01f1535fec3e_calibrate_947526_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bbe4e584-c2d3-4b33-a8d5-01f1535fec3e_calibrate_947526_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ce17b12d-3d6d-4c5e-b7f2-95ce4c80a0d2_characterizeImage_947526_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ce17b12d-3d6d-4c5e-b7f2-95ce4c80a0d2_characterizeImage_947526_22.stderr
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6138c27d96bcfa373ecd9eed64774c9 with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/83cf5273-d236-4c2d-aff5-4069d069f921_calibrate_947524_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/83cf5273-d236-4c2d-aff5-4069d069f921_calibrate_947524_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:32 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5d86e816ad926b7cc6d7eee61b893ee with result from task 713
parsl.dataflow.dflow DEBUG: Task 713 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e40fffed-db4c-4282-805a-cdfc335e97dc_calibrate_947523_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e40fffed-db4c-4282-805a-cdfc335e97dc_calibrate_947523_44.stderr
parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2355a907ef7d78480bbb7a95f40d4a5e with result from task 458
parsl.dataflow.dflow DEBUG: Task 458 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/62cc8b74-454f-4d70-951d-ab8480c71027_calibrate_947422_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/62cc8b74-454f-4d70-951d-ab8480c71027_calibrate_947422_10.stderr
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c820a42dc50661a1241ea67898fd0f5f with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d2f32d26-4a57-4e46-b4dd-3fa94447361b_calibrate_947524_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d2f32d26-4a57-4e46-b4dd-3fa94447361b_calibrate_947524_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:33 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41edc7ddc6702132372095425311638d with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ed1af37d-6757-4a41-99c2-a46092180131_calibrate_947420_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ed1af37d-6757-4a41-99c2-a46092180131_calibrate_947420_13.stderr
parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 564f194280f0b5eb132116275cd82ae3 with result from task 997
parsl.dataflow.dflow DEBUG: Task 997 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 37dc2a1350dca4d5c323acae7b774b71
parsl.dataflow.memoization INFO: Task 998 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1157
parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a5d9133f-969e-4bee-94b5-881404f5b5ee_calibrate_947418_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a5d9133f-969e-4bee-94b5-881404f5b5ee_calibrate_947418_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8cc4155c-21cb-489f-a2d4-9258ce35e250_characterizeImage_947418_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/8cc4155c-21cb-489f-a2d4-9258ce35e250_characterizeImage_947418_22.stderr
parsl.dataflow.dflow INFO: Task 1258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1e17e2e974d3dee155c46eb5d17f328 with result from task 1258
parsl.dataflow.dflow DEBUG: Task 1258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1259 has memoization hash d71444607b4604a4038d601dd5a87e82
parsl.dataflow.memoization INFO: Task 1259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1259 try 0 launched on executor multi with executor id 1158
parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c7571885-1a22-4723-b3cc-58c1122bdf80_calibrate_947422_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c7571885-1a22-4723-b3cc-58c1122bdf80_calibrate_947422_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0a6c5ffe-029d-4c3f-8b47-869ec0a94d61_characterizeImage_947422_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/0a6c5ffe-029d-4c3f-8b47-869ec0a94d61_characterizeImage_947422_42.stderr
parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 068f9d77aa25aed51298b022d993c704 with result from task 730
parsl.dataflow.dflow DEBUG: Task 730 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 731 has memoization hash c5268f2cf4fbfe3c928ed386bf0d66bf
parsl.dataflow.memoization INFO: Task 731 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e76480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 1159
parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/56d2248a-b881-4790-9412-3764fd0ebf4b_calibrate_947420_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/56d2248a-b881-4790-9412-3764fd0ebf4b_calibrate_947420_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/95e7f283-485f-4c71-abd1-1f11854b5806_characterizeImage_947420_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/95e7f283-485f-4c71-abd1-1f11854b5806_characterizeImage_947420_19.stderr
parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 45f255c998a6222252a4509846b51c4c with result from task 1049
parsl.dataflow.dflow DEBUG: Task 1049 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/0c7ea207-7e26-4b39-89bb-de908d7ddc41_calibrate_947420_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/0c7ea207-7e26-4b39-89bb-de908d7ddc41_calibrate_947420_57.stderr
parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e818bd3e4e6635fa513a6a33e41ede0b with result from task 1010
parsl.dataflow.dflow DEBUG: Task 1010 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/de3ae67b-2f68-4b54-bdee-2daff2263af8_calibrate_947420_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/de3ae67b-2f68-4b54-bdee-2daff2263af8_calibrate_947420_6.stderr
parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79c6e21ee28ea191501d6f67bbc021d1 with result from task 647
parsl.dataflow.dflow DEBUG: Task 647 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a372aec-cd86-42a6-b590-e5d811b3cd7c_calibrate_947422_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a372aec-cd86-42a6-b590-e5d811b3cd7c_calibrate_947422_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:33 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7bc512654d8ad6e2b35e86f076f997b with result from task 809
parsl.dataflow.dflow DEBUG: Task 809 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0687f110-cd2d-4119-8853-f98605d07bb3_calibrate_947523_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0687f110-cd2d-4119-8853-f98605d07bb3_calibrate_947523_33.stderr
parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c6d582d7fd9603676bff82be674b044 with result from task 422
parsl.dataflow.dflow DEBUG: Task 422 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/33daa05f-4304-42fa-85e1-4fe92f609572_calibrate_947418_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/33daa05f-4304-42fa-85e1-4fe92f609572_calibrate_947418_32.stderr
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4597cba808246903cf20c3f849bc55 with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/811f1078-a0e3-459a-8fd6-a520e4d76539_calibrate_947420_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/811f1078-a0e3-459a-8fd6-a520e4d76539_calibrate_947420_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:33 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b41ddc712cd2fc7496bad08f42de0c7 with result from task 974
parsl.dataflow.dflow DEBUG: Task 974 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/14cdcb45-a5b3-40e9-a024-1c8c350903ac_calibrate_947523_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/14cdcb45-a5b3-40e9-a024-1c8c350903ac_calibrate_947523_28.stderr
parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac40cd62bbc0de13de921e0536f96b83 with result from task 857
parsl.dataflow.dflow DEBUG: Task 857 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/4d665bf3-686b-4e20-8f5e-9c28f0818228_calibrate_947523_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/4d665bf3-686b-4e20-8f5e-9c28f0818228_calibrate_947523_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:34 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f52af58b802d5bd204f0182edd94d76 with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash c1b8168a8e54d6b92317b3792f3c9a7c
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e96c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 1160
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/19d291d3-0c38-41fb-80d6-90119f0790bb_calibrate_947524_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/19d291d3-0c38-41fb-80d6-90119f0790bb_calibrate_947524_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/23fe6b0c-b06a-4016-84fc-52a27373cb9b_characterizeImage_947524_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/23fe6b0c-b06a-4016-84fc-52a27373cb9b_characterizeImage_947524_54.stderr
parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 542b037d1e04a8ce828b1263d1e0dea6 with result from task 307
parsl.dataflow.dflow DEBUG: Task 307 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 662ec108f0cf94a8da3e6cd7baebcf27
parsl.dataflow.memoization INFO: Task 308 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 1161
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/55e4b08b-ea04-4772-8f32-5e51d2dde373_calibrate_947523_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/55e4b08b-ea04-4772-8f32-5e51d2dde373_calibrate_947523_22.stderr
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/67d57fc7-a298-45aa-9f70-9970f36f1530_characterizeImage_947523_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/67d57fc7-a298-45aa-9f70-9970f36f1530_characterizeImage_947523_22.stderr
parsl.dataflow.dflow INFO: Task 1222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 768ac637e39d93cd2f1cef93215a4292 with result from task 1222
parsl.dataflow.dflow DEBUG: Task 1222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1223 has memoization hash 25aaff1a3a9c72a113c52dea33d06aa9
parsl.dataflow.memoization INFO: Task 1223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1223 try 0 launched on executor multi with executor id 1162
parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/aa9266fc-f3c8-40f7-84ff-6dd0dd3f3088_calibrate_947523_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/aa9266fc-f3c8-40f7-84ff-6dd0dd3f3088_calibrate_947523_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/18521faf-71d9-459f-8348-9ec6092d8dc2_characterizeImage_947523_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/18521faf-71d9-459f-8348-9ec6092d8dc2_characterizeImage_947523_34.stderr
parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7fc15fb2841889351edde1e29094c640 with result from task 686
parsl.dataflow.dflow DEBUG: Task 686 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fdc83cf9-ef90-417f-a99b-6e467bff11e0_calibrate_947418_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fdc83cf9-ef90-417f-a99b-6e467bff11e0_calibrate_947418_31.stderr
parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 906eccbeb2076386a8e2681208ac5f6e with result from task 1105
parsl.dataflow.dflow DEBUG: Task 1105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash 455b15e0cfc1c345095b7f3d294c9418
parsl.dataflow.memoization INFO: Task 1106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 1163
parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f0b19d00-91f9-438f-9e60-72d5b243cee3_calibrate_947422_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f0b19d00-91f9-438f-9e60-72d5b243cee3_calibrate_947422_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ba07cd9b-3fe7-4318-a38e-8f5125b423ac_characterizeImage_947422_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/ba07cd9b-3fe7-4318-a38e-8f5125b423ac_characterizeImage_947422_43.stderr
parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8478bc04e71a4520db5d87457e60cbc9 with result from task 800
parsl.dataflow.dflow DEBUG: Task 800 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dcae169a-0851-42b7-bd92-07147e6daa3b_calibrate_947524_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dcae169a-0851-42b7-bd92-07147e6daa3b_calibrate_947524_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:34 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4059d85b6031b877050ee59b059ea96 with result from task 635
parsl.dataflow.dflow DEBUG: Task 635 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/217b5c10-c426-4bb6-82ad-898a624c61bc_calibrate_947418_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/217b5c10-c426-4bb6-82ad-898a624c61bc_calibrate_947418_45.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30c28cb3523ca12ca938bb415834e379 with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2b3b6618-87eb-4a8b-8c2b-80e8c152d6e8_calibrate_947523_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2b3b6618-87eb-4a8b-8c2b-80e8c152d6e8_calibrate_947523_32.stderr
parsl.dataflow.dflow INFO: Task 1234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d0c036e10511547b90f408da6e2a403 with result from task 1234
parsl.dataflow.dflow DEBUG: Task 1234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1235 has memoization hash e7a51ccee83513eccc71d24742498199
parsl.dataflow.memoization INFO: Task 1235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1235 try 0 launched on executor multi with executor id 1164
parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e19ba07f-ade1-4ba6-b8f2-efa544947e23_calibrate_947523_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e19ba07f-ade1-4ba6-b8f2-efa544947e23_calibrate_947523_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6a983877-8c63-4c21-8c11-09e6a92eefd8_characterizeImage_947523_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/6a983877-8c63-4c21-8c11-09e6a92eefd8_characterizeImage_947523_18.stderr
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae4bc0b92428c708785e507ef92801d with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/e18999b7-cb9d-4126-a82f-2f2fcd384a40_calibrate_947420_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/e18999b7-cb9d-4126-a82f-2f2fcd384a40_calibrate_947420_8.stderr
parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4517300e08685bf303be2df17ac998e6 with result from task 551
parsl.dataflow.dflow DEBUG: Task 551 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d08722c8-2055-4405-848b-1c1c9749df32_calibrate_947418_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/d08722c8-2055-4405-848b-1c1c9749df32_calibrate_947418_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:34 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2104c79c76e4cc7f6073865b10d6e2 with result from task 434
parsl.dataflow.dflow DEBUG: Task 434 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bacd90ef-3417-427e-95e7-0b2875fc2259_calibrate_947420_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bacd90ef-3417-427e-95e7-0b2875fc2259_calibrate_947420_45.stderr
parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41dda9a88216271970cc70766a813114 with result from task 326
parsl.dataflow.dflow DEBUG: Task 326 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/d93f4e49-c81b-4206-bb3b-0e89720a21d1_calibrate_947523_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/d93f4e49-c81b-4206-bb3b-0e89720a21d1_calibrate_947523_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:35 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd0a75d25722064a4bf335782d5a172c with result from task 1288
parsl.dataflow.dflow DEBUG: Task 1288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1289 has memoization hash e94d6aaa40bb5c6e7682cc922e91a04c
parsl.dataflow.memoization INFO: Task 1289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1289 try 0 launched on executor multi with executor id 1165
parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/cd95c9a8-521b-4ae1-8202-48947ee18636_calibrate_947523_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/cd95c9a8-521b-4ae1-8202-48947ee18636_calibrate_947523_14.stderr
parsl.dataflow.dflow INFO: Standard output for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3453b653-d8a1-4bb9-a2b2-4eac1e106db0_characterizeImage_947523_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/3453b653-d8a1-4bb9-a2b2-4eac1e106db0_characterizeImage_947523_14.stderr
parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4f442cb27db804025006ef9375f35ad with result from task 278
parsl.dataflow.dflow DEBUG: Task 278 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0bdf7a2d-8589-42ff-bdb4-e793d33db83a_calibrate_947523_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/0bdf7a2d-8589-42ff-bdb4-e793d33db83a_calibrate_947523_47.stderr
parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 905d106e83ee1fb755efb3dde71b5c7c with result from task 511
parsl.dataflow.dflow DEBUG: Task 511 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 8176101254207727b3358d6f32302e5b
parsl.dataflow.memoization INFO: Task 512 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 1166
parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8fc47b4d-8f34-4e26-865b-e4a13093de1f_calibrate_947528_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8fc47b4d-8f34-4e26-865b-e4a13093de1f_calibrate_947528_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/9d80d1a0-8fe8-43e4-b394-1765d68c0265_characterizeImage_947528_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/9d80d1a0-8fe8-43e4-b394-1765d68c0265_characterizeImage_947528_7.stderr
parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6762e1fc5e44fb068350b5ac5bb0f93 with result from task 863
parsl.dataflow.dflow DEBUG: Task 863 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ce5772ca-3b50-4562-b218-3f4d7702c5c6_calibrate_947420_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ce5772ca-3b50-4562-b218-3f4d7702c5c6_calibrate_947420_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:35 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:36 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9293184c3e88f28bacada7d62d177837 with result from task 952
parsl.dataflow.dflow DEBUG: Task 952 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 953 has memoization hash fb466ac17f813fc0c1461ebb5a2e22e3
parsl.dataflow.memoization INFO: Task 953 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 1167
parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a1da334-4ce9-4ee2-8340-6514909d15a0_calibrate_947523_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a1da334-4ce9-4ee2-8340-6514909d15a0_calibrate_947523_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5d4b3362-5ac0-4119-97ce-6bc36d4fddd9_characterizeImage_947523_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/5d4b3362-5ac0-4119-97ce-6bc36d4fddd9_characterizeImage_947523_60.stderr
parsl.dataflow.dflow INFO: Task 1135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35d6a97e0e54d8d26f92f6a38192a83a with result from task 1135
parsl.dataflow.dflow DEBUG: Task 1135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1136 has memoization hash c7d66b1c17ad8459f147a9615f33ede3
parsl.dataflow.memoization INFO: Task 1136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1136 try 0 launched on executor multi with executor id 1168
parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0fe78b18-69c8-4c01-b946-c7327a641796_calibrate_947422_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0fe78b18-69c8-4c01-b946-c7327a641796_calibrate_947422_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b521c164-b0a7-4aa2-9243-db6fb3b23e92_characterizeImage_947422_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b521c164-b0a7-4aa2-9243-db6fb3b23e92_characterizeImage_947422_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:36 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72271b45240caf79f84c4d4fe9e38806 with result from task 1186
parsl.dataflow.dflow DEBUG: Task 1186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1187 has memoization hash b71138f056c14c41810744481cd6469d
parsl.dataflow.memoization INFO: Task 1187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1187 try 0 launched on executor multi with executor id 1169
parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fb04abc3-1bac-44a8-9eb6-b25685d3a988_calibrate_947524_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fb04abc3-1bac-44a8-9eb6-b25685d3a988_calibrate_947524_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/488147ba-6dd6-4725-97dd-03ffae28e5d2_characterizeImage_947524_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/488147ba-6dd6-4725-97dd-03ffae28e5d2_characterizeImage_947524_35.stderr
parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98bc227224f0777b29cb1315fdb058bf with result from task 946
parsl.dataflow.dflow DEBUG: Task 946 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 947 has memoization hash 9055fffe59dc310403c4cd97d958be36
parsl.dataflow.memoization INFO: Task 947 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 1170
parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/275490cd-d264-411b-aa69-b15999710e9a_calibrate_947418_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/275490cd-d264-411b-aa69-b15999710e9a_calibrate_947418_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/21b1b8ec-3568-4946-8a4b-7b229cd385eb_characterizeImage_947418_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/21b1b8ec-3568-4946-8a4b-7b229cd385eb_characterizeImage_947418_8.stderr
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b3856175ad119786a83697c1093512d4 with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6cdccdd9-bc47-4ca7-8984-5fae969e5688_calibrate_947523_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6cdccdd9-bc47-4ca7-8984-5fae969e5688_calibrate_947523_6.stderr
parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry abc19fbfb6709871967685de9bdaa34c with result from task 263
parsl.dataflow.dflow DEBUG: Task 263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e014fcb1-a3c4-4cb3-8152-d458be2dceac_calibrate_947526_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e014fcb1-a3c4-4cb3-8152-d458be2dceac_calibrate_947526_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:36 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ba1b5b8cef8c69f999586a501360f92 with result from task 908
parsl.dataflow.dflow DEBUG: Task 908 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b0e9cb50-43e9-4dd3-8eb7-5e181d5d7f54_calibrate_947523_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b0e9cb50-43e9-4dd3-8eb7-5e181d5d7f54_calibrate_947523_15.stderr
parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e467dbefba4e5109ed380cba4a4f0112 with result from task 758
parsl.dataflow.dflow DEBUG: Task 758 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1f83c109-6bec-47ca-9443-c5e1fff7c859_calibrate_947523_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/1f83c109-6bec-47ca-9443-c5e1fff7c859_calibrate_947523_40.stderr
parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c33888ba954eb638488ac2ca3b419f96 with result from task 431
parsl.dataflow.dflow DEBUG: Task 431 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b25dde45-0a95-489a-8c63-f5b09e94973e_calibrate_947422_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b25dde45-0a95-489a-8c63-f5b09e94973e_calibrate_947422_37.stderr
parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13f16a203b04f3df42bd42eeecf47264 with result from task 607
parsl.dataflow.dflow DEBUG: Task 607 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 608 has memoization hash c1690744ee9684ae01a7c1ac7631761f
parsl.dataflow.memoization INFO: Task 608 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 1171
parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7d34ade7-0908-4c73-bbfa-a6f18075fac2_calibrate_947418_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7d34ade7-0908-4c73-bbfa-a6f18075fac2_calibrate_947418_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7a53135e-bc27-4380-8058-0f156dcb8d35_characterizeImage_947418_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/7a53135e-bc27-4380-8058-0f156dcb8d35_characterizeImage_947418_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:37 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2506e7e0dbea666f339cc9ceec680fb7 with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 6728c2cf94c7c7e9c771e30b3d841dcd
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa467740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 1172
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c1378830-4e1a-4fe4-a07e-6e6e1f65dd44_calibrate_947422_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c1378830-4e1a-4fe4-a07e-6e6e1f65dd44_calibrate_947422_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f988318d-eacf-4d02-a10b-25b88f52a56f_characterizeImage_947422_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f988318d-eacf-4d02-a10b-25b88f52a56f_characterizeImage_947422_40.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63bd681e5e343371b1ebe28aadc3bdd7 with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ffe281bc-8be1-4cc4-8ef9-09c0c4a772de_calibrate_947420_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ffe281bc-8be1-4cc4-8ef9-09c0c4a772de_calibrate_947420_27.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0dbd07a3560fd0f311c332ef256b71a2 with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/56aba7cf-65a4-46ed-907c-cc781c72cb62_calibrate_947418_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/56aba7cf-65a4-46ed-907c-cc781c72cb62_calibrate_947418_53.stderr
parsl.dataflow.dflow INFO: Task 1243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3179ed1264cf69cc9adb7f37d9f7815e with result from task 1243
parsl.dataflow.dflow DEBUG: Task 1243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1244 has memoization hash a0492478b4edf95a71c5105d9bb02afc
parsl.dataflow.memoization INFO: Task 1244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1244 try 0 launched on executor multi with executor id 1173
parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d8e8983d-9746-4125-82cf-b60bd5c947dc_calibrate_947528_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d8e8983d-9746-4125-82cf-b60bd5c947dc_calibrate_947528_43.stderr
parsl.dataflow.dflow INFO: Standard output for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/51ff2bf5-648c-43e7-b4df-f66c1352a9df_characterizeImage_947528_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/51ff2bf5-648c-43e7-b4df-f66c1352a9df_characterizeImage_947528_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:37 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc49e8b69ea9accdab3c7e369e4e5efa with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 9711be7c769544d39207620781f0bbed
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 1174
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/69e9aa9a-5735-4614-8e22-8db95ca1bd74_calibrate_947524_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/69e9aa9a-5735-4614-8e22-8db95ca1bd74_calibrate_947524_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/63030db4-caaa-4b82-8981-0880ed3973fe_characterizeImage_947524_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/63030db4-caaa-4b82-8981-0880ed3973fe_characterizeImage_947524_24.stderr
parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0949c9aef05ff716f0dee27f5763fa92 with result from task 740
parsl.dataflow.dflow DEBUG: Task 740 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b2ec7803-34dd-451e-8b89-16d3bc29702f_calibrate_947422_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b2ec7803-34dd-451e-8b89-16d3bc29702f_calibrate_947422_28.stderr
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ac6518c821cd4bda4059a47d6946b71c with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 33aec193b3bbb80f5cbae712273e70ae
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e97240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 1175
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bffd4049-1b00-4983-8ad0-6422c0ca0d29_calibrate_947526_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bffd4049-1b00-4983-8ad0-6422c0ca0d29_calibrate_947526_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b9c1ecd9-ce10-478d-a033-0c9c455f353c_characterizeImage_947526_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/b9c1ecd9-ce10-478d-a033-0c9c455f353c_characterizeImage_947526_40.stderr
parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 632c22e3fb69c55fb0b6fab64febf280 with result from task 964
parsl.dataflow.dflow DEBUG: Task 964 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 4905fad1fcb48ea5721eef633a797004
parsl.dataflow.memoization INFO: Task 965 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 1176
parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/848aff7f-9b50-4007-b7e0-e2bd3e057bc3_calibrate_947523_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/848aff7f-9b50-4007-b7e0-e2bd3e057bc3_calibrate_947523_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/df1ffd64-003d-4d12-9e21-ebd7bba867de_characterizeImage_947523_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/df1ffd64-003d-4d12-9e21-ebd7bba867de_characterizeImage_947523_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:38 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7466c6efd843a7a354f454b3b26a7ecd with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fe845a2e-3b97-4f19-a6fd-932dffb6fc9b_calibrate_947524_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fe845a2e-3b97-4f19-a6fd-932dffb6fc9b_calibrate_947524_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:38 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d88510f0ba6d6ea5ff8fe71579dee4f7 with result from task 1141
parsl.dataflow.dflow DEBUG: Task 1141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1142 has memoization hash 80aab61449ac0e29f12229325f5104ac
parsl.dataflow.memoization INFO: Task 1142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1142 try 0 launched on executor multi with executor id 1177
parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/89d0cbe6-f8ad-4a56-922a-84eb6b491f7f_calibrate_947528_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/89d0cbe6-f8ad-4a56-922a-84eb6b491f7f_calibrate_947528_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/499f322b-ae0e-4700-b07a-76bbeea1c985_characterizeImage_947528_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/499f322b-ae0e-4700-b07a-76bbeea1c985_characterizeImage_947528_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:38 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e1fc90671eaa0a7864bd7f408fb5249 with result from task 188
parsl.dataflow.dflow DEBUG: Task 188 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ad3ca06d-6099-4dbd-93cc-962cfb8ff8d1_calibrate_947420_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ad3ca06d-6099-4dbd-93cc-962cfb8ff8d1_calibrate_947420_4.stderr
parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68031a07ba26ea1cc9af78b699f19292 with result from task 593
parsl.dataflow.dflow DEBUG: Task 593 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/896815f6-0b8c-4ec5-aecd-7ec6615dc61e_calibrate_947422_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/896815f6-0b8c-4ec5-aecd-7ec6615dc61e_calibrate_947422_13.stderr
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3e7398a404d573bead796942502fbc7 with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 7e96682c632d9f93f7140e20b2b0aab4
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 1178
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5da5ba1a-df94-413e-b1b8-b1499daaff76_calibrate_947420_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5da5ba1a-df94-413e-b1b8-b1499daaff76_calibrate_947420_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/917a10a4-bfcd-4892-95c1-5e357dcffdb7_characterizeImage_947420_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/917a10a4-bfcd-4892-95c1-5e357dcffdb7_characterizeImage_947420_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:39 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b31860103a049d5af619ba1e0e5f365 with result from task 274
parsl.dataflow.dflow DEBUG: Task 274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 681db473c8fe851acdbe22c49bee069c
parsl.dataflow.memoization INFO: Task 275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 1179
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/206c951a-c85e-4980-bb7d-eaa4feab28d1_calibrate_947524_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/206c951a-c85e-4980-bb7d-eaa4feab28d1_calibrate_947524_53.stderr
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d86f5e71-54e6-4069-ba5d-23c516302c95_characterizeImage_947524_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d86f5e71-54e6-4069-ba5d-23c516302c95_characterizeImage_947524_53.stderr
parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26469e31da9ce9e10d05d234eb263fc4 with result from task 1018
parsl.dataflow.dflow DEBUG: Task 1018 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 4f4347734a3f503f453d5bb514f89534
parsl.dataflow.memoization INFO: Task 1019 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 1180
parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/73741d9f-58af-4110-b126-42107d5fda32_calibrate_947420_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/73741d9f-58af-4110-b126-42107d5fda32_calibrate_947420_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4e9185c4-5fba-4ee0-9bfe-fcf6660e1c94_characterizeImage_947420_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/4e9185c4-5fba-4ee0-9bfe-fcf6660e1c94_characterizeImage_947420_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:39 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ffffa400dadfa9522d61bf98ab675de3 with result from task 988
parsl.dataflow.dflow DEBUG: Task 988 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 989 has memoization hash 2aa7fc1c399290ab60fc18603cd487de
parsl.dataflow.memoization INFO: Task 989 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 1181
parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ba97eb2d-d4ce-40f9-a913-9e17a60b78ae_calibrate_947420_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ba97eb2d-d4ce-40f9-a913-9e17a60b78ae_calibrate_947420_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/c6f13e59-55b0-4c18-9b13-c38a3629baed_characterizeImage_947420_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/c6f13e59-55b0-4c18-9b13-c38a3629baed_characterizeImage_947420_9.stderr
parsl.dataflow.dflow INFO: Task 1298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db784fa5c7c43afbb142ab6511f828fc with result from task 1298
parsl.dataflow.dflow DEBUG: Task 1298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/88364500-088b-4f0e-91d9-5879e2739c6f_calibrate_947523_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/88364500-088b-4f0e-91d9-5879e2739c6f_calibrate_947523_43.stderr
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9405ebac9d37ff1fabe96b07ca35432 with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/2792f5a7-6c4d-4e4a-9bac-5cca371f9c65_calibrate_947526_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/2792f5a7-6c4d-4e4a-9bac-5cca371f9c65_calibrate_947526_13.stderr
parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d82c0d7ed7346a41c28d909efc7b77b1 with result from task 212
parsl.dataflow.dflow DEBUG: Task 212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5abdbbb4-89b8-443e-aa20-4c3e22070600_calibrate_947418_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5abdbbb4-89b8-443e-aa20-4c3e22070600_calibrate_947418_41.stderr
parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5686900b77c05dc9663a16b0b3baf33a with result from task 671
parsl.dataflow.dflow DEBUG: Task 671 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15c07d76-2af2-48fc-b77a-8b5d1eb4fced_calibrate_947420_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15c07d76-2af2-48fc-b77a-8b5d1eb4fced_calibrate_947420_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:40 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f68527a2efbf9f233ba1883c75b69cac with result from task 1204
parsl.dataflow.dflow DEBUG: Task 1204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1205 has memoization hash 548e2d352438db7b8d041e0e2169fb31
parsl.dataflow.memoization INFO: Task 1205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1205 try 0 launched on executor multi with executor id 1182
parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3786ea10-43dc-4f3f-b4c1-3a5d24b4a1a0_calibrate_947420_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3786ea10-43dc-4f3f-b4c1-3a5d24b4a1a0_calibrate_947420_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e7c4c10c-f0fa-47f6-90c0-fd1be9425810_characterizeImage_947420_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/e7c4c10c-f0fa-47f6-90c0-fd1be9425810_characterizeImage_947420_37.stderr
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebcf9323595f1a39180dbc63b0083e07 with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fb6daf65-b1bf-405e-91c0-21192091e7ee_calibrate_947418_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fb6daf65-b1bf-405e-91c0-21192091e7ee_calibrate_947418_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:40 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f25c36a3802e8c14f2c35ed61bfd755b with result from task 556
parsl.dataflow.dflow DEBUG: Task 556 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 557 has memoization hash c71243149308846fff4bbbe5001b4404
parsl.dataflow.memoization INFO: Task 557 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 1183
parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/542e1ac6-8ea7-41aa-bc28-f32e93348243_calibrate_947418_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/542e1ac6-8ea7-41aa-bc28-f32e93348243_calibrate_947418_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ae3b2afa-9e3e-4997-b57a-38033d50cd65_characterizeImage_947418_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/ae3b2afa-9e3e-4997-b57a-38033d50cd65_characterizeImage_947418_5.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0a14218bb2e4fdba6ab1ca6bcf49fc6 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1019c02d-bbba-4d80-b922-9c079508a8bc_calibrate_947418_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/1019c02d-bbba-4d80-b922-9c079508a8bc_calibrate_947418_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:40 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28f7e6c180d35989cbad943b1aec7282 with result from task 218
parsl.dataflow.dflow DEBUG: Task 218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/aa86da19-c3d6-472f-952b-93e83efee985_calibrate_947418_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/aa86da19-c3d6-472f-952b-93e83efee985_calibrate_947418_47.stderr
parsl.dataflow.dflow INFO: Task 1267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15e4056fdc34c7db05493aad4fbdeb3f with result from task 1267
parsl.dataflow.dflow DEBUG: Task 1267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1268 has memoization hash a226239893fea9a5a845544bc2057383
parsl.dataflow.memoization INFO: Task 1268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1268 try 0 launched on executor multi with executor id 1184
parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1d1966ac-253e-4646-8417-1958ca6a130c_calibrate_947528_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1d1966ac-253e-4646-8417-1958ca6a130c_calibrate_947528_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cf2411d9-6a0b-4775-bb89-5f45a153413f_characterizeImage_947528_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/cf2411d9-6a0b-4775-bb89-5f45a153413f_characterizeImage_947528_48.stderr
parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35d1df0161341e08bd9728298769d639 with result from task 248
parsl.dataflow.dflow DEBUG: Task 248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/c90e5d0a-9bd3-4496-845d-f37c1976eaec_calibrate_947420_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/c90e5d0a-9bd3-4496-845d-f37c1976eaec_calibrate_947420_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:41 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 97ad4d5bd94ef5da3edc7a934a8b4e54 with result from task 398
parsl.dataflow.dflow DEBUG: Task 398 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/669a1001-a4d6-476d-9df1-c87d76105e83_calibrate_947420_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/669a1001-a4d6-476d-9df1-c87d76105e83_calibrate_947420_34.stderr
parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51de3fcdefc7dedf796247f9f74d759b with result from task 1001
parsl.dataflow.dflow DEBUG: Task 1001 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/01f2c3c7-b53a-4e69-b92e-f62c474de019_calibrate_947422_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/01f2c3c7-b53a-4e69-b92e-f62c474de019_calibrate_947422_15.stderr
parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca1f35fa4f52fa6318c32c5bf8f92839 with result from task 386
parsl.dataflow.dflow DEBUG: Task 386 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/573593c0-4ade-4dde-aab2-19ec4f60916c_calibrate_947422_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/573593c0-4ade-4dde-aab2-19ec4f60916c_calibrate_947422_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:41 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df5be1dda22ff2aa4c12e9ea8bbec001 with result from task 982
parsl.dataflow.dflow DEBUG: Task 982 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 983 has memoization hash d4a66dd08bdb38983fcb92e5cf5cb6c0
parsl.dataflow.memoization INFO: Task 983 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 1185
parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d74bddb5-e2fc-4b9c-b108-30c952796a9f_calibrate_947420_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d74bddb5-e2fc-4b9c-b108-30c952796a9f_calibrate_947420_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/540f099a-1a67-4db7-a973-5caee3fb823f_characterizeImage_947420_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/540f099a-1a67-4db7-a973-5caee3fb823f_characterizeImage_947420_29.stderr
parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8d559d6a30e3432af226799a8864c35 with result from task 1015
parsl.dataflow.dflow DEBUG: Task 1015 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash ffb081c7588a367c0f48ebe34a504705
parsl.dataflow.memoization INFO: Task 1016 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1186
parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/077104cc-8ffc-4820-aa15-140b64ecfc78_calibrate_947420_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/077104cc-8ffc-4820-aa15-140b64ecfc78_calibrate_947420_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/31ba8635-6c06-444d-aea4-be0e3faa3c70_characterizeImage_947420_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/31ba8635-6c06-444d-aea4-be0e3faa3c70_characterizeImage_947420_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:42 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64273d1ea8e62f46bff09b5230b5b3fb with result from task 919
parsl.dataflow.dflow DEBUG: Task 919 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 920 has memoization hash eb9cfe2702316702b82bcd56b9f47f5b
parsl.dataflow.memoization INFO: Task 920 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 1187
parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8916b295-0512-4031-9840-59b28de38aae_calibrate_947422_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8916b295-0512-4031-9840-59b28de38aae_calibrate_947422_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/06544cc6-1c0c-4a69-b463-68286ab9f1de_characterizeImage_947422_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/06544cc6-1c0c-4a69-b463-68286ab9f1de_characterizeImage_947422_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:42 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 794b798d2626f76cc612ea75f1d06020 with result from task 295
parsl.dataflow.dflow DEBUG: Task 295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 9df2f3a4a1c84ef1eea1bbe48f94336a
parsl.dataflow.memoization INFO: Task 296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 1188
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f90c8c0-54b7-495f-b702-3eb1ef40d25a_calibrate_947523_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f90c8c0-54b7-495f-b702-3eb1ef40d25a_calibrate_947523_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/60900f45-0e04-4e53-910a-916fde56226e_characterizeImage_947523_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/60900f45-0e04-4e53-910a-916fde56226e_characterizeImage_947523_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:42 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 574d308631b722e7897bbee107822866 with result from task 250
parsl.dataflow.dflow DEBUG: Task 250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 0eae2efecf34ab4c00edd9951a231748
parsl.dataflow.memoization INFO: Task 251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 1189
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1c64395e-2d56-4a60-a986-442d376c2996_calibrate_947528_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1c64395e-2d56-4a60-a986-442d376c2996_calibrate_947528_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1fab9ce6-d5ab-41f6-b036-c902fb7774a9_characterizeImage_947528_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1fab9ce6-d5ab-41f6-b036-c902fb7774a9_characterizeImage_947528_16.stderr
parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccafd9e2efc59a520427bff88ffa7f1a with result from task 311
parsl.dataflow.dflow DEBUG: Task 311 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a311ae4c-a558-4bb3-8933-1d97f826ffce_calibrate_947418_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a311ae4c-a558-4bb3-8933-1d97f826ffce_calibrate_947418_3.stderr
parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c4715ef35231e8db6ca1022e14fe257 with result from task 1067
parsl.dataflow.dflow DEBUG: Task 1067 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9a62b401-bddd-4290-b65f-542bdc9c5cc5_calibrate_947524_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9a62b401-bddd-4290-b65f-542bdc9c5cc5_calibrate_947524_5.stderr
parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13eee97f0cf99a6ec9a6d02b87fc41e9 with result from task 1112
parsl.dataflow.dflow DEBUG: Task 1112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5d05ffca-9f24-4442-8cb8-371d68adad7d_calibrate_947418_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/5d05ffca-9f24-4442-8cb8-371d68adad7d_calibrate_947418_50.stderr
parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 366078ce6a319e2bb7c9bdcdb3da61bf with result from task 986
parsl.dataflow.dflow DEBUG: Task 986 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b8d5f24-1c73-4ed0-bfee-ab8412b94ecf_calibrate_947523_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b8d5f24-1c73-4ed0-bfee-ab8412b94ecf_calibrate_947523_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:43 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9661a7c1c540eabb9efbc78f63560736 with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/7307ed97-04b3-4e12-b1cb-53eb43e5b994_calibrate_947523_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/7307ed97-04b3-4e12-b1cb-53eb43e5b994_calibrate_947523_36.stderr
parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c946e80274c9e3c49f9ed11c976e7e39 with result from task 683
parsl.dataflow.dflow DEBUG: Task 683 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b994f19f-9243-4037-b8e6-8b682156feee_calibrate_947528_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b994f19f-9243-4037-b8e6-8b682156feee_calibrate_947528_62.stderr
parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 028caa3c15625281839eb75dfecf3514 with result from task 763
parsl.dataflow.dflow DEBUG: Task 763 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 3fc3aa85ea09150e5f684e19a286016e
parsl.dataflow.memoization INFO: Task 764 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 1190
parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0c3fa93a-10a6-4150-be9b-ae23d4e63a85_calibrate_947528_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0c3fa93a-10a6-4150-be9b-ae23d4e63a85_calibrate_947528_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6c115c9b-d7bf-4d94-8dab-a708c81941bc_characterizeImage_947528_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6c115c9b-d7bf-4d94-8dab-a708c81941bc_characterizeImage_947528_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:43 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f802e52cc135b3b784bc5a178d68319 with result from task 364
parsl.dataflow.dflow DEBUG: Task 364 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 96fd4717f4b49fce4b8ce7ee6505a93f
parsl.dataflow.memoization INFO: Task 365 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 1191
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/19c6dd54-14aa-4316-841e-f54f46fd49f8_calibrate_947420_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/19c6dd54-14aa-4316-841e-f54f46fd49f8_calibrate_947420_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/327dd411-0946-4cb1-bf57-73fbc1d98353_characterizeImage_947420_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/327dd411-0946-4cb1-bf57-73fbc1d98353_characterizeImage_947420_30.stderr
parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a01388059596a972d0cdd243f97fd417 with result from task 419
parsl.dataflow.dflow DEBUG: Task 419 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b56f5108-90db-4d45-87d7-eb6200484bda_calibrate_947528_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b56f5108-90db-4d45-87d7-eb6200484bda_calibrate_947528_32.stderr
parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5581209b615ffc3498f7e1084ca81e57 with result from task 298
parsl.dataflow.dflow DEBUG: Task 298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 8aba59ec035e3c7dee5f9ad4d977b7d0
parsl.dataflow.memoization INFO: Task 299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 1192
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/49f6e3a1-47a3-4d18-9aad-cc954243b823_calibrate_947526_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/49f6e3a1-47a3-4d18-9aad-cc954243b823_calibrate_947526_55.stderr
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/254358e0-3a00-4887-be59-8258df35c665_characterizeImage_947526_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/254358e0-3a00-4887-be59-8258df35c665_characterizeImage_947526_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:43 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f85b5c28dc41ef6e2019d689c54c20fc with result from task 695
parsl.dataflow.dflow DEBUG: Task 695 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b8435979-a08f-4dd7-b984-2c0ee5a4c4e7_calibrate_947422_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/b8435979-a08f-4dd7-b984-2c0ee5a4c4e7_calibrate_947422_4.stderr
parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2667fdcc48a6285f60aeb0a6e9c102 with result from task 266
parsl.dataflow.dflow DEBUG: Task 266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/79a3bde7-8b71-4eef-bbc3-ff0399da738c_calibrate_947526_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/79a3bde7-8b71-4eef-bbc3-ff0399da738c_calibrate_947526_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:44 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85d979d4e39b9462dda9ce600c54f072 with result from task 1094
parsl.dataflow.dflow DEBUG: Task 1094 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/6b03b0f9-df96-40d9-bac9-4ec6850419c7_calibrate_947420_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/6b03b0f9-df96-40d9-bac9-4ec6850419c7_calibrate_947420_54.stderr
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f52133c384507fdb19e7b724ca405b80 with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash f6b02605df3d3f56f84bd3cb5b4edcde
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 1193
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d144c03a-6c83-4526-9504-6f6eff420907_calibrate_947524_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d144c03a-6c83-4526-9504-6f6eff420907_calibrate_947524_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/551690df-ce1f-459c-a693-627391d96a30_characterizeImage_947524_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/551690df-ce1f-459c-a693-627391d96a30_characterizeImage_947524_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:44 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfe6e8b90c831e4e6a835de578727254 with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 2e17f0a7819761b2a56fff1b51534186
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 1194
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/27561a78-0b5d-4f12-97d9-05f17c91ed2f_calibrate_947526_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/27561a78-0b5d-4f12-97d9-05f17c91ed2f_calibrate_947526_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/4eddb58d-0f24-4009-af06-56641fd1af49_characterizeImage_947526_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/4eddb58d-0f24-4009-af06-56641fd1af49_characterizeImage_947526_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:44 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1970737f32fef91ebc57fa3e212ed595 with result from task 1036
parsl.dataflow.dflow DEBUG: Task 1036 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 3f0b4bdbbc969b9efcba08ed738b3416
parsl.dataflow.memoization INFO: Task 1037 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1195
parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/5b1fe438-0815-4455-b973-73a1b9ecd706_calibrate_947524_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/5b1fe438-0815-4455-b973-73a1b9ecd706_calibrate_947524_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7cfd55d0-fa93-4df0-9d5c-23d4c7714085_characterizeImage_947524_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7cfd55d0-fa93-4df0-9d5c-23d4c7714085_characterizeImage_947524_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:45 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca0e91efdac085888ae83762cd202e4d with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f6785c32-23be-4aad-bff3-8b7ac5ee5c1b_calibrate_947523_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f6785c32-23be-4aad-bff3-8b7ac5ee5c1b_calibrate_947523_51.stderr
parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12b25f8a2b82fcf5a59bad56afa3b0cb with result from task 535
parsl.dataflow.dflow DEBUG: Task 535 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 536 has memoization hash fcd73849f285cd882359d8c31a0b523f
parsl.dataflow.memoization INFO: Task 536 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 1196
parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/12457ad7-3453-4387-a875-f0561db89018_calibrate_947526_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/12457ad7-3453-4387-a875-f0561db89018_calibrate_947526_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/7a5a6818-80d5-4e51-b0f2-b85af1725bd0_characterizeImage_947526_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/7a5a6818-80d5-4e51-b0f2-b85af1725bd0_characterizeImage_947526_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:45 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8f2c5ce10d60656ea42fca41130e95a with result from task 569
parsl.dataflow.dflow DEBUG: Task 569 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9168b06b-b297-461b-b06d-627a748a9d27_calibrate_947420_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/9168b06b-b297-461b-b06d-627a748a9d27_calibrate_947420_12.stderr
parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bcd60ff1227261ddbed05d6616b48fb5 with result from task 725
parsl.dataflow.dflow DEBUG: Task 725 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5ddcb4fd-f85f-4be0-b672-aa4311cb0f2a_calibrate_947526_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5ddcb4fd-f85f-4be0-b672-aa4311cb0f2a_calibrate_947526_53.stderr
parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5909e0509a76147e6ebfeb055280790 with result from task 932
parsl.dataflow.dflow DEBUG: Task 932 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ca29e3a5-f9a1-4dc3-aead-7140cec38064_calibrate_947523_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/ca29e3a5-f9a1-4dc3-aead-7140cec38064_calibrate_947523_39.stderr
parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1a62d768e8fa878089990070d2db04 with result from task 1088
parsl.dataflow.dflow DEBUG: Task 1088 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/59aedb6d-eac8-4e69-8779-0c445e5ea7e6_calibrate_947528_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/59aedb6d-eac8-4e69-8779-0c445e5ea7e6_calibrate_947528_9.stderr
parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a1458753c0e9c43bfbdf87bce9cc6af8 with result from task 1076
parsl.dataflow.dflow DEBUG: Task 1076 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9213d8f4-df36-44e9-bd79-ec4be670f505_calibrate_947422_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9213d8f4-df36-44e9-bd79-ec4be670f505_calibrate_947422_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:45 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93ca71e254b199f326562a856795cd0e with result from task 961
parsl.dataflow.dflow DEBUG: Task 961 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 7bea4dd1d3a8db23d9e4ec77199ceb9a
parsl.dataflow.memoization INFO: Task 962 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 1197
parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/97bfa387-69c5-4a87-a4a6-286a46ca081d_calibrate_947528_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/97bfa387-69c5-4a87-a4a6-286a46ca081d_calibrate_947528_42.stderr
parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/59bfa5b2-ac34-4741-8a19-ffd8c60b869f_characterizeImage_947528_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/59bfa5b2-ac34-4741-8a19-ffd8c60b869f_characterizeImage_947528_42.stderr
parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cae56f5c7337f80ea355e54d58e9165c with result from task 833
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:46 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow DEBUG: Task 833 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/660d3e08-aa2b-4b18-b5bc-646432fd4e60_calibrate_947418_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/660d3e08-aa2b-4b18-b5bc-646432fd4e60_calibrate_947418_52.stderr
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a52c2451b1f759165846fa0b562db912 with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/39076b8e-9d22-40d2-886c-334870e19df7_calibrate_947418_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/39076b8e-9d22-40d2-886c-334870e19df7_calibrate_947418_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:46 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8778565c95cfe07128d07ee26808fe4 with result from task 1052
parsl.dataflow.dflow DEBUG: Task 1052 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/7e8629a8-6bd6-492f-a7ef-33458866faa9_calibrate_947528_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/7e8629a8-6bd6-492f-a7ef-33458866faa9_calibrate_947528_49.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:46 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91b73f75083f57cae8c1fc3b8d6c9564 with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/dd5497f1-2aa3-4458-b2eb-6ad8c50b60a9_calibrate_947420_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/dd5497f1-2aa3-4458-b2eb-6ad8c50b60a9_calibrate_947420_60.stderr
parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec43d5fe6bb83cf0fb63d482d047af20 with result from task 532
parsl.dataflow.dflow DEBUG: Task 532 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 81959b16ce6d66eb39200b40b4efad6d
parsl.dataflow.memoization INFO: Task 533 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 1198
parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d784cef2-f283-4b9b-b813-adf64a359d9f_calibrate_947528_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d784cef2-f283-4b9b-b813-adf64a359d9f_calibrate_947528_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/698887ce-cf85-43cb-8431-fa1e65dc8d80_characterizeImage_947528_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/698887ce-cf85-43cb-8431-fa1e65dc8d80_characterizeImage_947528_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:47 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b720215cc4ca6915481abbaefeedf1b3 with result from task 782
parsl.dataflow.dflow DEBUG: Task 782 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/1fb7ac68-db64-4a7f-a64b-ab54c23ebb04_calibrate_947422_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/1fb7ac68-db64-4a7f-a64b-ab54c23ebb04_calibrate_947422_44.stderr
parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa6e1713d4e55610b628c076b83a0f91 with result from task 347
parsl.dataflow.dflow DEBUG: Task 347 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8d5ddaaa-5842-4ed6-adf2-9b310588c043_calibrate_947524_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8d5ddaaa-5842-4ed6-adf2-9b310588c043_calibrate_947524_33.stderr
parsl.dataflow.dflow INFO: Task 1189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c4973aa3d07fcee982c08a53e419e5a with result from task 1189
parsl.dataflow.dflow DEBUG: Task 1189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1190 has memoization hash b8b4dd52814a45fb518c32b57d577dd4
parsl.dataflow.memoization INFO: Task 1190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1190 try 0 launched on executor multi with executor id 1199
parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/7f528d70-9737-43b2-9527-dfea17eeca70_calibrate_947422_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/7f528d70-9737-43b2-9527-dfea17eeca70_calibrate_947422_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/867a1dfd-2e4c-438d-9b8c-669392b8088b_characterizeImage_947422_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/867a1dfd-2e4c-438d-9b8c-669392b8088b_characterizeImage_947422_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:47 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 140f77dc2deb961ca9644db26ad48b87 with result from task 851
parsl.dataflow.dflow DEBUG: Task 851 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/54710d40-45dc-4192-a776-a12c66cbd770_calibrate_947523_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/54710d40-45dc-4192-a776-a12c66cbd770_calibrate_947523_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:47 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3503e1bc6620d0cc872fe958c2fa5c8 with result from task 689
parsl.dataflow.dflow DEBUG: Task 689 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e452d87e-b9fb-4699-b4e1-c328f8f8ffdd_calibrate_947526_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e452d87e-b9fb-4699-b4e1-c328f8f8ffdd_calibrate_947526_31.stderr
parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd84d28f5461375f018ea0c3827df06 with result from task 1024
parsl.dataflow.dflow DEBUG: Task 1024 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 5ab38dc7c617c123a37fa9277674bbbd
parsl.dataflow.memoization INFO: Task 1025 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1200
parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e2fc2082-07a3-42dc-b894-2f9974fca693_calibrate_947418_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e2fc2082-07a3-42dc-b894-2f9974fca693_calibrate_947418_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e2eca7fd-864a-46d4-9092-4b1ddf72e76b_characterizeImage_947418_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e2eca7fd-864a-46d4-9092-4b1ddf72e76b_characterizeImage_947418_20.stderr
parsl.dataflow.dflow INFO: Task 1183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 718c17ef747d45e582d3e36056ba135b with result from task 1183
parsl.dataflow.dflow DEBUG: Task 1183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1184 has memoization hash 5970084c48affbeef951d319a1198aea
parsl.dataflow.memoization INFO: Task 1184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1184 try 0 launched on executor multi with executor id 1201
parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/23c0d55d-937c-4e40-9ab4-c133e76b7c43_calibrate_947528_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/23c0d55d-937c-4e40-9ab4-c133e76b7c43_calibrate_947528_15.stderr
parsl.dataflow.dflow INFO: Standard output for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e87b88c-4c81-499c-a0e2-7deeaeba5faa_characterizeImage_947528_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e87b88c-4c81-499c-a0e2-7deeaeba5faa_characterizeImage_947528_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:47 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2a9e80baaa7361d142614015cbc1b41 with result from task 869
parsl.dataflow.dflow DEBUG: Task 869 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/fb8fe7c5-87f3-43b5-8832-13f4a63e8a37_calibrate_947526_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/fb8fe7c5-87f3-43b5-8832-13f4a63e8a37_calibrate_947526_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:47 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc564eebea3c900ce072df3912c4589f with result from task 1133
parsl.dataflow.dflow DEBUG: Task 1133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a8bf300e-b4bb-4c23-b0bf-06da135c9b5a_calibrate_947418_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a8bf300e-b4bb-4c23-b0bf-06da135c9b5a_calibrate_947418_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 695c8e43d10ca70d5232241e1833b22c with result from task 1240
parsl.dataflow.dflow DEBUG: Task 1240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1241 has memoization hash c142547641a580b44ba182eabfea0009
parsl.dataflow.memoization INFO: Task 1241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1241 try 0 launched on executor multi with executor id 1202
parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c804e15c-24a6-43ea-829a-80b3a4e9442e_calibrate_947523_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c804e15c-24a6-43ea-829a-80b3a4e9442e_calibrate_947523_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/37814657-a34e-452c-8205-a6f48f920804_characterizeImage_947523_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/37814657-a34e-452c-8205-a6f48f920804_characterizeImage_947523_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7c10d8c7f94b0112f5f3d03a70c019e with result from task 658
parsl.dataflow.dflow DEBUG: Task 658 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 21c778eb628220eef62309c7a5244cac
parsl.dataflow.memoization INFO: Task 659 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e35940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 1203
parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e3b459f2-ff9d-402f-8e1d-ec9f956cb1b8_calibrate_947526_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e3b459f2-ff9d-402f-8e1d-ec9f956cb1b8_calibrate_947526_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/85cf8048-f25e-4974-bbfc-b754b0eb9e3c_characterizeImage_947526_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/85cf8048-f25e-4974-bbfc-b754b0eb9e3c_characterizeImage_947526_56.stderr
parsl.dataflow.dflow INFO: Task 1246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85de3f78e33b04569acbd9efd52a4de6 with result from task 1246
parsl.dataflow.dflow DEBUG: Task 1246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1247 has memoization hash 8a650f6e796f7d3bb3cd18ea1757ccaf
parsl.dataflow.memoization INFO: Task 1247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1247 try 0 launched on executor multi with executor id 1204
parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/895238d0-000a-42a7-9cd2-e39c5177c1ea_calibrate_947418_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/895238d0-000a-42a7-9cd2-e39c5177c1ea_calibrate_947418_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/774b9d4f-3b32-483a-a801-d683e94f59da_characterizeImage_947418_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/774b9d4f-3b32-483a-a801-d683e94f59da_characterizeImage_947418_28.stderr
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e264bcc4bbde49a67e151e3713db4822 with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a32f9ab-126f-4ff0-972c-6f9093c2eb00_calibrate_947523_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5a32f9ab-126f-4ff0-972c-6f9093c2eb00_calibrate_947523_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:48 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09297afee3905e1674b11fbd2fc6be94 with result from task 787
parsl.dataflow.dflow DEBUG: Task 787 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 3f4baa6aa4b731c9c7c2198c8353ce6c
parsl.dataflow.memoization INFO: Task 788 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 1205
parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/3e177025-4ecd-490a-88d5-0efac89cabdf_calibrate_947524_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/3e177025-4ecd-490a-88d5-0efac89cabdf_calibrate_947524_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b9af7cab-ac59-4341-b232-a6372f924947_characterizeImage_947524_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b9af7cab-ac59-4341-b232-a6372f924947_characterizeImage_947524_16.stderr
parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8462bb97902c51cfa02002f806ff987 with result from task 415
parsl.dataflow.dflow DEBUG: Task 415 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 416 has memoization hash bc0da4fdfd79631f575fe6e525ed7fdd
parsl.dataflow.memoization INFO: Task 416 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 1206
parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/1bb804de-a880-4d34-9dd4-67b4b0b3c93c_calibrate_947526_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/1bb804de-a880-4d34-9dd4-67b4b0b3c93c_calibrate_947526_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fd4b92dc-0d49-4325-9416-0f96d5f3af7a_characterizeImage_947526_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fd4b92dc-0d49-4325-9416-0f96d5f3af7a_characterizeImage_947526_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b5f814cf11dd56c8a76876616fd461f with result from task 775
parsl.dataflow.dflow DEBUG: Task 775 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 3b61812b3d9533b82d95e9760942a37d
parsl.dataflow.memoization INFO: Task 776 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 1207
parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/916a2f2b-c6ec-44d5-9a09-fa2dbf80aa77_calibrate_947528_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/916a2f2b-c6ec-44d5-9a09-fa2dbf80aa77_calibrate_947528_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5d2dec1f-4ba6-4878-8813-fe61947c835f_characterizeImage_947528_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5d2dec1f-4ba6-4878-8813-fe61947c835f_characterizeImage_947528_58.stderr
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f0176b8e0a3b643d9f6d1255f23c044 with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/677a761e-1198-4c4c-a170-f850c107069e_calibrate_947528_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/677a761e-1198-4c4c-a170-f850c107069e_calibrate_947528_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0a0f347374850cdd4034a17fd7d4083 with result from task 409
parsl.dataflow.dflow DEBUG: Task 409 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 53a8a7812d2c6d0a01aa4f1de71c0722
parsl.dataflow.memoization INFO: Task 410 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f51300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 1208
parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e5e41e8a-0ac6-4e1b-8d17-75edd607de78_calibrate_947526_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e5e41e8a-0ac6-4e1b-8d17-75edd607de78_calibrate_947526_28.stderr
parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/e8f6f792-0ac3-4183-98c6-a05fba58b40d_characterizeImage_947526_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/e8f6f792-0ac3-4183-98c6-a05fba58b40d_characterizeImage_947526_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01204d060f8c1561a9f6172f74b42656 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f264cca0-504d-47c3-bb57-f49d3e160664_calibrate_947523_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f264cca0-504d-47c3-bb57-f49d3e160664_calibrate_947523_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 675ee48fe7b8f2f76277a94d904c3d98 with result from task 1150
parsl.dataflow.dflow DEBUG: Task 1150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1151 has memoization hash 063161a879270c6b4e6623f6728e977c
parsl.dataflow.memoization INFO: Task 1151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1151 try 0 launched on executor multi with executor id 1209
parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/33034da0-8562-4b6b-b1b4-2afd6e59168d_calibrate_947526_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/33034da0-8562-4b6b-b1b4-2afd6e59168d_calibrate_947526_48.stderr
parsl.dataflow.dflow INFO: Standard output for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8e5aaa63-e492-406f-9052-5834b75313ce_characterizeImage_947526_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/8e5aaa63-e492-406f-9052-5834b75313ce_characterizeImage_947526_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:49 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98fa8ba238bf5b32bb085a930ebbf6ed with result from task 697
parsl.dataflow.dflow DEBUG: Task 697 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 8ab6b1f2196c78892790cb9a43bc538c
parsl.dataflow.memoization INFO: Task 698 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e74400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 1210
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7fc7f8b9-6b16-46a2-a09c-f3b11ddffbd7_calibrate_947526_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7fc7f8b9-6b16-46a2-a09c-f3b11ddffbd7_calibrate_947526_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08ec80a8-fa50-4649-a42d-ac7cab71aeae_characterizeImage_947526_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08ec80a8-fa50-4649-a42d-ac7cab71aeae_characterizeImage_947526_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33f44775b4e73cf9619c1ffa6ac7ff68 with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/25f7e71f-48e1-4b9c-bb9f-f1c36202cd1d_calibrate_947528_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/25f7e71f-48e1-4b9c-bb9f-f1c36202cd1d_calibrate_947528_26.stderr
parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5850851e0538140dc7be7efa85d8fa0e with result from task 928
parsl.dataflow.dflow DEBUG: Task 928 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 929 has memoization hash bb928f56a20baa0bbecebd0c7cf2c2ac
parsl.dataflow.memoization INFO: Task 929 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 1211
parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e549ef4d-f234-4cda-a56b-6f7d537a473a_calibrate_947523_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e549ef4d-f234-4cda-a56b-6f7d537a473a_calibrate_947523_56.stderr
parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/90fc6dfa-9260-4425-bf98-3ffa5d033733_characterizeImage_947523_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/90fc6dfa-9260-4425-bf98-3ffa5d033733_characterizeImage_947523_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f4982153afe963884ddd2c989591847c with result from task 529
parsl.dataflow.dflow DEBUG: Task 529 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 15d81b1ef038676313f472f3248ca225
parsl.dataflow.memoization INFO: Task 530 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 1212
parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5576eb-5a89-4a22-b826-97f48c82538b_calibrate_947526_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5576eb-5a89-4a22-b826-97f48c82538b_calibrate_947526_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d807620e-4b1b-44b2-b2c9-2d1454648b70_characterizeImage_947526_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d807620e-4b1b-44b2-b2c9-2d1454648b70_characterizeImage_947526_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:50 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8cc598c4c86f3402125b24cc589af82 with result from task 1117
parsl.dataflow.dflow DEBUG: Task 1117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1118 has memoization hash 4f28ccc3436eb93f194fe77094ad5de0
parsl.dataflow.memoization INFO: Task 1118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1118 try 0 launched on executor multi with executor id 1213
parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0501a906-bc15-410b-bd1b-28c882ce2057_calibrate_947422_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0501a906-bc15-410b-bd1b-28c882ce2057_calibrate_947422_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d9c8f0e7-9a07-464b-9a57-d6e27ac6eff3_characterizeImage_947422_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/d9c8f0e7-9a07-464b-9a57-d6e27ac6eff3_characterizeImage_947422_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c0e4d06c605e9934a0d9e2572c4ed8f with result from task 208
parsl.dataflow.dflow DEBUG: Task 208 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 4451fec005ef3803f2b6c27e1cf95fbb
parsl.dataflow.memoization INFO: Task 209 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 1214
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae6cb67-17d4-4a16-a280-c85ec376dbd0_calibrate_947524_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae6cb67-17d4-4a16-a280-c85ec376dbd0_calibrate_947524_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/80796b49-aa4a-41ec-8529-b0f52e0ae278_characterizeImage_947524_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/80796b49-aa4a-41ec-8529-b0f52e0ae278_characterizeImage_947524_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62a348973cbac841570a2d5fbeee04a4 with result from task 509
parsl.dataflow.dflow DEBUG: Task 509 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ee67882-1098-4530-a027-d2cb326c8141_calibrate_947526_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ee67882-1098-4530-a027-d2cb326c8141_calibrate_947526_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:51 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e089b94b68be68bcc3b26cec5ed356b with result from task 383
parsl.dataflow.dflow DEBUG: Task 383 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/31458834-a4d2-4e91-bd2d-a5c5bb18556f_calibrate_947420_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/31458834-a4d2-4e91-bd2d-a5c5bb18556f_calibrate_947420_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:52 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8072a201c5a74b67c60336c2d6c9f509 with result from task 497
parsl.dataflow.dflow DEBUG: Task 497 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9b271258-0352-4981-982c-4277515f9b76_calibrate_947523_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9b271258-0352-4981-982c-4277515f9b76_calibrate_947523_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8da15ab05e6048ad5b42255aa06f65e with result from task 940
parsl.dataflow.dflow DEBUG: Task 940 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 6a87878dedd3150deeb795fe12d19cd0
parsl.dataflow.memoization INFO: Task 941 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 1215
parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/6165cf3f-2cd5-4a1e-b15a-aac4d716347f_calibrate_947524_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/6165cf3f-2cd5-4a1e-b15a-aac4d716347f_calibrate_947524_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4d433184-603d-44b5-8e24-89c75163c292_characterizeImage_947524_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/4d433184-603d-44b5-8e24-89c75163c292_characterizeImage_947524_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 550e3c3176644d271013ce3b865d95e9 with result from task 499
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 499 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 1e92bdb5646fa4ed2fb6250072d7a4a9
parsl.dataflow.memoization INFO: Task 500 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d87100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 1216
parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6fd0a284-2ef1-492b-883b-5da46a1ddf13_calibrate_947523_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6fd0a284-2ef1-492b-883b-5da46a1ddf13_calibrate_947523_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a9c06fef-a94b-4e67-8f79-5f91fcd505fd_characterizeImage_947523_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/a9c06fef-a94b-4e67-8f79-5f91fcd505fd_characterizeImage_947523_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8026edbccd87cdf0e05f56135fde3468 with result from task 1276
parsl.dataflow.dflow DEBUG: Task 1276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1277 has memoization hash 648472687166fbcc241dd7ab5f60eda1
parsl.dataflow.memoization INFO: Task 1277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1277 try 0 launched on executor multi with executor id 1217
parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/6e10fd72-dd0f-4ebc-a49d-3d50f639ec7a_calibrate_947422_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/6e10fd72-dd0f-4ebc-a49d-3d50f639ec7a_calibrate_947422_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a813b9df-3966-4b1a-bb93-3b7f19f3ec80_characterizeImage_947422_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/a813b9df-3966-4b1a-bb93-3b7f19f3ec80_characterizeImage_947422_30.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc10d105e2e8114a6b3e77728d88fa1b with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 052d9585a9d325085d113234f7f37a37
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 1218
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0ac4511d-2cc6-4ecd-b946-a60f1ea814df_calibrate_947526_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0ac4511d-2cc6-4ecd-b946-a60f1ea814df_calibrate_947526_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b51cd98-1124-4008-887f-5373be0c9ad4_characterizeImage_947526_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2b51cd98-1124-4008-887f-5373be0c9ad4_characterizeImage_947526_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:53 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d64e4509957893fc69b0a3b42e42a6c2 with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f2e4f294-addb-4b9b-b596-2c4c1fb7e0c0_calibrate_947523_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f2e4f294-addb-4b9b-b596-2c4c1fb7e0c0_calibrate_947523_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:54 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f29c3b6889655b1df5ec2c46930430f with result from task 617
parsl.dataflow.dflow DEBUG: Task 617 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a82ef996-a77e-41fa-bf38-e991e570cb82_calibrate_947526_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a82ef996-a77e-41fa-bf38-e991e570cb82_calibrate_947526_9.stderr
parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c81b70c6c982233e4fead57ec531c12f with result from task 611
parsl.dataflow.dflow DEBUG: Task 611 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f6d8b962-e523-4c12-9674-be8d9b753118_calibrate_947526_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f6d8b962-e523-4c12-9674-be8d9b753118_calibrate_947526_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:54 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42963f90099c19be1238597e294761c8 with result from task 893
parsl.dataflow.dflow DEBUG: Task 893 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/02dba8bf-b8b0-41cc-a7d9-4faded03d67c_calibrate_947422_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/02dba8bf-b8b0-41cc-a7d9-4faded03d67c_calibrate_947422_14.stderr
parsl.dataflow.dflow INFO: Task 1213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b871d15babc509616f94ac0b5a2d0f24 with result from task 1213
parsl.dataflow.dflow DEBUG: Task 1213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1214 has memoization hash 7945bd772cde824b3e021f755740ac19
parsl.dataflow.memoization INFO: Task 1214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1214 try 0 launched on executor multi with executor id 1219
parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/319ef1be-964c-4d67-9e60-8b5d777b1838_calibrate_947524_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/319ef1be-964c-4d67-9e60-8b5d777b1838_calibrate_947524_59.stderr
parsl.dataflow.dflow INFO: Standard output for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/33a9a86c-182b-45e1-b84d-625303ec5c0d_characterizeImage_947524_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 1213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/33a9a86c-182b-45e1-b84d-625303ec5c0d_characterizeImage_947524_59.stderr
parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0c0684c0c877a89d993ea62160c6dcc with result from task 641
parsl.dataflow.dflow DEBUG: Task 641 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5efd9047-a85e-4137-98e7-f7cbb8609c60_calibrate_947528_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5efd9047-a85e-4137-98e7-f7cbb8609c60_calibrate_947528_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:54 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f54a8e1c3360a8a743811186ddb02ce4 with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 9b0028b46246dc0627565839955df762
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 1220
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/74d07640-cf14-4990-88de-2aabeeb6c4e3_calibrate_947524_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/74d07640-cf14-4990-88de-2aabeeb6c4e3_calibrate_947524_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d98895c3-5024-4011-b618-b255cf25ce4d_characterizeImage_947524_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/d98895c3-5024-4011-b618-b255cf25ce4d_characterizeImage_947524_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:55 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry adc7af201627e0b1f337301e17fd8336 with result from task 1177
parsl.dataflow.dflow DEBUG: Task 1177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1178 has memoization hash 576c54bb72343ccd377c64f426554228
parsl.dataflow.memoization INFO: Task 1178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1178 try 0 launched on executor multi with executor id 1221
parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/020ad163-32fc-4b2b-af8b-1959a08f765b_calibrate_947526_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/020ad163-32fc-4b2b-af8b-1959a08f765b_calibrate_947526_34.stderr
parsl.dataflow.dflow INFO: Standard output for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec62d6c9-ae9a-430c-b2b5-d3f9207d9af7_characterizeImage_947526_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec62d6c9-ae9a-430c-b2b5-d3f9207d9af7_characterizeImage_947526_34.stderr
parsl.dataflow.dflow INFO: Task 1165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aeb1a13e37c212e07e4135cb537a36cb with result from task 1165
parsl.dataflow.dflow DEBUG: Task 1165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1166 has memoization hash cd3393deaad9c929e82e1a7c7cdc2bfb
parsl.dataflow.memoization INFO: Task 1166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1166 try 0 launched on executor multi with executor id 1222
parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/36beb3db-f089-45a3-b1e4-118f5e9ba596_calibrate_947526_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/36beb3db-f089-45a3-b1e4-118f5e9ba596_calibrate_947526_23.stderr
parsl.dataflow.dflow INFO: Standard output for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08bdd8da-a653-41d2-9d0a-73d5646b5a14_characterizeImage_947526_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/08bdd8da-a653-41d2-9d0a-73d5646b5a14_characterizeImage_947526_23.stderr
parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae3d38de4e3fee33280a9cd7a27ce728 with result from task 830
parsl.dataflow.dflow DEBUG: Task 830 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/7a1ebbbb-8d12-465f-94e7-b66ff1a516f7_calibrate_947420_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/7a1ebbbb-8d12-465f-94e7-b66ff1a516f7_calibrate_947420_38.stderr
parsl.dataflow.dflow INFO: Task 1217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88053e20539613fe002a95148439b646 with result from task 1217
parsl.dataflow.dflow DEBUG: Task 1217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79f375ca-4805-43f9-a5b1-95e9a74bb56c_calibrate_947418_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 1217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79f375ca-4805-43f9-a5b1-95e9a74bb56c_calibrate_947418_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:55 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8268debf20b79a677af55da0f5062225 with result from task 1279
parsl.dataflow.dflow DEBUG: Task 1279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1280 has memoization hash 26ad97da7ff91a424ad928300ad66e0b
parsl.dataflow.memoization INFO: Task 1280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1280 try 0 launched on executor multi with executor id 1223
parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15afa65a-7887-45d8-a61d-f736de01eb02_calibrate_947420_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15afa65a-7887-45d8-a61d-f736de01eb02_calibrate_947420_44.stderr
parsl.dataflow.dflow INFO: Standard output for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af7e7348-0694-4f26-ada6-03bb4dcac92b_characterizeImage_947420_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/af7e7348-0694-4f26-ada6-03bb4dcac92b_characterizeImage_947420_44.stderr
parsl.dataflow.dflow INFO: Task 1201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59248b5439a711f4cb578ff3bb7cec2e with result from task 1201
parsl.dataflow.dflow DEBUG: Task 1201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1202 has memoization hash fab8834d14ec8121129b43454d1cc47b
parsl.dataflow.memoization INFO: Task 1202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1202 try 0 launched on executor multi with executor id 1224
parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5705427f-2dba-476c-aacf-f4f79289fad2_calibrate_947528_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5705427f-2dba-476c-aacf-f4f79289fad2_calibrate_947528_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5021c100-2883-4c19-8811-f101929ea69c_characterizeImage_947528_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/5021c100-2883-4c19-8811-f101929ea69c_characterizeImage_947528_57.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df2c58d352e419c115a0b14779890445 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1bc934ca-0c07-4c6a-9411-93dcf3992a5b_calibrate_947524_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1bc934ca-0c07-4c6a-9411-93dcf3992a5b_calibrate_947524_44.stderr
parsl.dataflow.dflow INFO: Task 1124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 782a67ebeb0f11a2fc2f6d2c909ee1c0 with result from task 1124
parsl.dataflow.dflow DEBUG: Task 1124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f0182558-4b38-4ec4-a387-8173b1a68395_calibrate_947418_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/f0182558-4b38-4ec4-a387-8173b1a68395_calibrate_947418_23.stderr
parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b71b0d8ddcc53d5f06b5a9eac577ed7 with result from task 886
parsl.dataflow.dflow DEBUG: Task 886 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 46851a11774bd0910274c0dd5bb944ca
parsl.dataflow.memoization INFO: Task 887 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 1225
parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9c037f8a-1b44-4cc8-ae95-1e5810a648b8_calibrate_947422_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9c037f8a-1b44-4cc8-ae95-1e5810a648b8_calibrate_947422_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/89df2ad1-0c3a-4ca5-b25c-2b8a0f4c4da6_characterizeImage_947422_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/89df2ad1-0c3a-4ca5-b25c-2b8a0f4c4da6_characterizeImage_947422_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:56 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fd2368f8eacd3da5a3fa9bb10b215a4 with result from task 944
parsl.dataflow.dflow DEBUG: Task 944 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/4df6149b-8c27-47ad-bf4a-ef98ecb64581_calibrate_947528_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/4df6149b-8c27-47ad-bf4a-ef98ecb64581_calibrate_947528_51.stderr
parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c577a869b6d71d3801cf966dc5a7edc4 with result from task 443
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:56 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow DEBUG: Task 443 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d402039c-4688-46b4-919a-c65701699f21_calibrate_947528_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d402039c-4688-46b4-919a-c65701699f21_calibrate_947528_21.stderr
parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7ac30ed7457747c554b4203be802a22 with result from task 362
parsl.dataflow.dflow DEBUG: Task 362 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f96f2f8b-db2d-4851-bdea-7838156fed1e_calibrate_947420_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/f96f2f8b-db2d-4851-bdea-7838156fed1e_calibrate_947420_11.stderr
parsl.dataflow.dflow INFO: Task 1195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68818b3105b61c739c37709c7047cbd8 with result from task 1195
parsl.dataflow.dflow DEBUG: Task 1195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1196 has memoization hash c79a91adde1a323b5e900f533b14700f
parsl.dataflow.memoization INFO: Task 1196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1196 try 0 launched on executor multi with executor id 1226
parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/368771bf-d69e-433a-9b88-b23f197c4323_calibrate_947523_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/368771bf-d69e-433a-9b88-b23f197c4323_calibrate_947523_54.stderr
parsl.dataflow.dflow INFO: Standard output for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2d659a02-c77d-4596-a10f-06cd204411c9_characterizeImage_947523_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/2d659a02-c77d-4596-a10f-06cd204411c9_characterizeImage_947523_54.stderr
parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab488e4961886b898a1f176c2b76b8f0 with result from task 301
parsl.dataflow.dflow DEBUG: Task 301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 5eecf8333057051e0131e5b4c454693c
parsl.dataflow.memoization INFO: Task 302 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f02200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 1227
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b0a5c6e5-b220-4364-841e-0f11ef28230f_calibrate_947526_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b0a5c6e5-b220-4364-841e-0f11ef28230f_calibrate_947526_12.stderr
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/771e82d6-4c13-4b18-b7c1-040e307c695b_characterizeImage_947526_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/771e82d6-4c13-4b18-b7c1-040e307c695b_characterizeImage_947526_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:28:56 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 186d99b783f5d78cab95f4240652a1f7 with result from task 1286
parsl.dataflow.dflow DEBUG: Task 1286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a1a86e92-11de-4494-9465-c17ac00945f4_calibrate_947526_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 1286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a1a86e92-11de-4494-9465-c17ac00945f4_calibrate_947526_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 248
parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 248
parsl.dataflow.strategy DEBUG: Executor multi has 248 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d25263a6050e3a1a5978d9ac648f12f5 with result from task 518
parsl.dataflow.dflow DEBUG: Task 518 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2aed7856-786b-446d-af11-5d80f7c6229e_calibrate_947523_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/2aed7856-786b-446d-af11-5d80f7c6229e_calibrate_947523_24.stderr
parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47506b7d9bf4116965b89bd51520ea17 with result from task 1042
parsl.dataflow.dflow DEBUG: Task 1042 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 51882b9a1125bf994603027584414b40
parsl.dataflow.memoization INFO: Task 1043 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 1228
parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d8a86879-7497-448a-9d78-a7354f5bc447_calibrate_947526_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d8a86879-7497-448a-9d78-a7354f5bc447_calibrate_947526_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ba68bde8-5dd2-4c13-b4c3-0fa09ec9e512_characterizeImage_947526_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ba68bde8-5dd2-4c13-b4c3-0fa09ec9e512_characterizeImage_947526_17.stderr
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f040c0ec54fe1e149fd1821305a74b0 with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5dd3f740-6b44-4ade-b5e0-c75a41f58968_calibrate_947526_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/5dd3f740-6b44-4ade-b5e0-c75a41f58968_calibrate_947526_14.stderr
parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1c946856d36ca08a0157ad395925973 with result from task 821
parsl.dataflow.dflow DEBUG: Task 821 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/58cb3447-fc8f-42b1-a6d9-63306a2173c9_calibrate_947526_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/58cb3447-fc8f-42b1-a6d9-63306a2173c9_calibrate_947526_15.stderr
parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 858177e26e8ec74f7734061752eb756a with result from task 824
parsl.dataflow.dflow DEBUG: Task 824 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/34b80b96-2409-4a52-8244-f566e070f366_calibrate_947528_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/34b80b96-2409-4a52-8244-f566e070f366_calibrate_947528_37.stderr
parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 717b7ccd08f94743a1514ec654e00336 with result from task 562
parsl.dataflow.dflow DEBUG: Task 562 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 19adb1388f86c9b3f77f479c95fab4e3
parsl.dataflow.memoization INFO: Task 563 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8db2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 1229
parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f2dea5e2-0a5f-47f3-8553-5f0cb2ceb844_calibrate_947528_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f2dea5e2-0a5f-47f3-8553-5f0cb2ceb844_calibrate_947528_36.stderr
parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fa68a04b-7c81-4832-b923-feb86d23e519_characterizeImage_947528_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/fa68a04b-7c81-4832-b923-feb86d23e519_characterizeImage_947528_36.stderr
parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00ee52ea7a0db1f95cea1bbd8c22748f with result from task 575
parsl.dataflow.dflow DEBUG: Task 575 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fd50a89f-d1b6-4293-b0dc-a7608606ddb1_calibrate_947524_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fd50a89f-d1b6-4293-b0dc-a7608606ddb1_calibrate_947524_29.stderr
parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72d862c35d6e458fe01e00e0a8e6476e with result from task 644
parsl.dataflow.dflow DEBUG: Task 644 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e167c6b1-59b5-4874-9cce-8d5611454024_calibrate_947524_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e167c6b1-59b5-4874-9cce-8d5611454024_calibrate_947524_49.stderr
parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9db9c407317894f42f5dddca909c48d6 with result from task 1114
parsl.dataflow.dflow DEBUG: Task 1114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 47cb65d3d287cd5476b353d4f25db99f
parsl.dataflow.memoization INFO: Task 1115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1230
parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ca27478-5124-42df-8e63-6718b8e10b3f_calibrate_947526_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ca27478-5124-42df-8e63-6718b8e10b3f_calibrate_947526_7.stderr
parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec944901-3a17-4893-9c25-9ad129d24bd7_characterizeImage_947526_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ec944901-3a17-4893-9c25-9ad129d24bd7_characterizeImage_947526_7.stderr
parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0760abbbe6f858111d0d6e04307cad2e with result from task 404
parsl.dataflow.dflow DEBUG: Task 404 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b0b4355-425f-4011-b705-788bf9247086_calibrate_947523_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6b0b4355-425f-4011-b705-788bf9247086_calibrate_947523_11.stderr
parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37dc2a1350dca4d5c323acae7b774b71 with result from task 998
parsl.dataflow.dflow DEBUG: Task 998 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a5d9133f-969e-4bee-94b5-881404f5b5ee_calibrate_947418_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/a5d9133f-969e-4bee-94b5-881404f5b5ee_calibrate_947418_22.stderr
parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10ed70bb29431029b3871b780ba832ec with result from task 1109
parsl.dataflow.dflow DEBUG: Task 1109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c7ab8636-3ec7-400a-a51e-31d1a2b3f6d4_calibrate_947418_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/c7ab8636-3ec7-400a-a51e-31d1a2b3f6d4_calibrate_947418_40.stderr
parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8df0f6b8531380101b7f62ed80de2468 with result from task 1102
parsl.dataflow.dflow DEBUG: Task 1102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash e61721f296e81805897a2d1fb2f87ac7
parsl.dataflow.memoization INFO: Task 1103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c36980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 1231
parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/caf3eda1-2c8a-43f3-8937-90764862d38c_calibrate_947422_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/caf3eda1-2c8a-43f3-8937-90764862d38c_calibrate_947422_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b407d6c6-2f4d-47c8-a650-4d16dbab7199_characterizeImage_947422_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/b407d6c6-2f4d-47c8-a650-4d16dbab7199_characterizeImage_947422_3.stderr
parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44678e3ad0a8ffd2230c5c2472eb6ca4 with result from task 745
parsl.dataflow.dflow DEBUG: Task 745 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 2aeb1acd881931c90b7c35ab88a677e2
parsl.dataflow.memoization INFO: Task 746 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 1232
parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a134df97-00d6-4165-bff3-76012a50f696_calibrate_947524_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a134df97-00d6-4165-bff3-76012a50f696_calibrate_947524_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/daf5e1b3-c944-4897-b39e-4df43b29002f_characterizeImage_947524_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/daf5e1b3-c944-4897-b39e-4df43b29002f_characterizeImage_947524_45.stderr
parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba43e62c35349cdba8188ab54bde64cf with result from task 925
parsl.dataflow.dflow DEBUG: Task 925 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 926 has memoization hash 8449fcb09194f5cda944a1385fa93afc
parsl.dataflow.memoization INFO: Task 926 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 1233
parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/746cd0a9-25b2-4480-bdd7-ec073e1157d2_calibrate_947524_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/746cd0a9-25b2-4480-bdd7-ec073e1157d2_calibrate_947524_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1cd4ac1a-d24e-4b16-a981-894176d99a4a_characterizeImage_947524_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1cd4ac1a-d24e-4b16-a981-894176d99a4a_characterizeImage_947524_6.stderr
parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe6c89ed3bcd58f8247a712df1067e17 with result from task 778
parsl.dataflow.dflow DEBUG: Task 778 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 7539cde4920e785540958a01a3e74e3f
parsl.dataflow.memoization INFO: Task 779 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8cc56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 1234
parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b3f12e7b-5056-4d39-a231-92f29a9a1787_calibrate_947523_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b3f12e7b-5056-4d39-a231-92f29a9a1787_calibrate_947523_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4455ea7c-b94a-492c-bb91-afb4597743d2_characterizeImage_947523_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4455ea7c-b94a-492c-bb91-afb4597743d2_characterizeImage_947523_61.stderr
parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26efee371aad96a3294ade0927c5491f with result from task 905
parsl.dataflow.dflow DEBUG: Task 905 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e7fdd721-25b4-450c-ab62-2e280868c753_calibrate_947422_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e7fdd721-25b4-450c-ab62-2e280868c753_calibrate_947422_41.stderr
parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50e8839ce009417778d1ab892bc7d676 with result from task 794
parsl.dataflow.dflow DEBUG: Task 794 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/549c0d14-eefa-40b9-8100-a1abe20a15a7_calibrate_947422_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/549c0d14-eefa-40b9-8100-a1abe20a15a7_calibrate_947422_19.stderr
parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6eeea93e7497f05092ee951ddaaa5234 with result from task 392
parsl.dataflow.dflow DEBUG: Task 392 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/08ca40b8-e876-406f-8af6-10dd5ce2f1f6_calibrate_947420_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/08ca40b8-e876-406f-8af6-10dd5ce2f1f6_calibrate_947420_22.stderr
parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e00fa91acd15c71c84744982525360f with result from task 605
parsl.dataflow.dflow DEBUG: Task 605 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae73baa-69ec-4166-836a-cc3bb21385e9_calibrate_947524_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae73baa-69ec-4166-836a-cc3bb21385e9_calibrate_947524_62.stderr
parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 61afa25196d2ff80c59fdaf73bcc2703 with result from task 395
parsl.dataflow.dflow DEBUG: Task 395 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/98bc0e40-dc29-4ef0-9f33-2db0ab702fb9_calibrate_947418_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/98bc0e40-dc29-4ef0-9f33-2db0ab702fb9_calibrate_947418_42.stderr
parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 402121cfbcf84075e0804fe20ec6ac91 with result from task 1003
parsl.dataflow.dflow DEBUG: Task 1003 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash a52dfecd7f158f234b1dacb4a3276253
parsl.dataflow.memoization INFO: Task 1004 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1235
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d8d11901-37e1-4436-8832-a980d6f79b78_calibrate_947524_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d8d11901-37e1-4436-8832-a980d6f79b78_calibrate_947524_3.stderr
parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/3aa34a38-02d1-4507-8394-90c5fb5a4500_characterizeImage_947524_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/3aa34a38-02d1-4507-8394-90c5fb5a4500_characterizeImage_947524_3.stderr
parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 207dd3e742e9107406a8a97781e89faa with result from task 1064
parsl.dataflow.dflow DEBUG: Task 1064 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f348d33-bc9b-432b-8ebb-dd7d1d1dc375_calibrate_947528_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f348d33-bc9b-432b-8ebb-dd7d1d1dc375_calibrate_947528_3.stderr
parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae1d244b425b2c681eea7213a49660b9 with result from task 743
parsl.dataflow.dflow DEBUG: Task 743 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8454de94-ae04-4bf0-a681-b26726fd00bc_calibrate_947524_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/8454de94-ae04-4bf0-a681-b26726fd00bc_calibrate_947524_7.stderr
parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4fb98d31c0935280ed9c2b1e3bbfe17 with result from task 587
parsl.dataflow.dflow DEBUG: Task 587 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/273e9131-ebfd-4787-ae0e-44fabb44fc47_calibrate_947523_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/273e9131-ebfd-4787-ae0e-44fabb44fc47_calibrate_947523_1.stderr
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2be245fb249d16b0e7a1ec6e2d266ce with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/300f8f1f-e5d7-4f6f-80e5-533a29e11fc7_calibrate_947528_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/300f8f1f-e5d7-4f6f-80e5-533a29e11fc7_calibrate_947528_34.stderr
parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 031bbc2f22c96c623d123b97083cfe7f with result from task 752
parsl.dataflow.dflow DEBUG: Task 752 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f1c852d2-dfc5-4b88-a982-097fbadf3b83_calibrate_947422_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f1c852d2-dfc5-4b88-a982-097fbadf3b83_calibrate_947422_22.stderr
parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bec4339be055e5b78e260e99c38e1f16 with result from task 473
parsl.dataflow.dflow DEBUG: Task 473 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1ae12f04-8718-42b6-b315-44d3e4c689b9_calibrate_947524_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/1ae12f04-8718-42b6-b315-44d3e4c689b9_calibrate_947524_27.stderr
parsl.dataflow.dflow INFO: Task 1159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16a89b54f2590a690247d476f591ba3c with result from task 1159
parsl.dataflow.dflow DEBUG: Task 1159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1160 has memoization hash 8028f1a903ec7518b8f19863a60e1c90
parsl.dataflow.memoization INFO: Task 1160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1160 try 0 launched on executor multi with executor id 1236
parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/3cbd35f9-af8c-4d12-ac4a-77a97f6b4d72_calibrate_947528_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/3cbd35f9-af8c-4d12-ac4a-77a97f6b4d72_calibrate_947528_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6e7d9729-fe29-4ae4-bb51-ab0ab1e1633d_characterizeImage_947528_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/6e7d9729-fe29-4ae4-bb51-ab0ab1e1633d_characterizeImage_947528_29.stderr
parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f06667f51cfd9919696d3d2da15a33bb with result from task 707
parsl.dataflow.dflow DEBUG: Task 707 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f9f4e8dd-481a-4200-9552-6ade6dfb2e66_calibrate_947422_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f9f4e8dd-481a-4200-9552-6ade6dfb2e66_calibrate_947422_32.stderr
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77323de4c27aaf69795915ad1fda9d93 with result from task 980
parsl.dataflow.dflow DEBUG: Task 980 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f826f325-17aa-47c7-a38f-a66226a0fa63_calibrate_947528_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f826f325-17aa-47c7-a38f-a66226a0fa63_calibrate_947528_50.stderr
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52c4bfce8bb97ce95388bd71210a7a23 with result from task 476
parsl.dataflow.dflow DEBUG: Task 476 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/cf86ca05-2bff-4b71-8cdd-ce531580cb39_calibrate_947526_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/cf86ca05-2bff-4b71-8cdd-ce531580cb39_calibrate_947526_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:02 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d7feed9d71f93cc6b524b97c0812254 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a3b2731-984a-4ec8-9a85-77efa0423a0c_calibrate_947422_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5a3b2731-984a-4ec8-9a85-77efa0423a0c_calibrate_947422_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57c0683d0c860ffc322cd1f1bd344496 with result from task 806
parsl.dataflow.dflow DEBUG: Task 806 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b6fddb5a-eccd-4bee-8374-15211bbeeb1d_calibrate_947528_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b6fddb5a-eccd-4bee-8374-15211bbeeb1d_calibrate_947528_40.stderr
parsl.dataflow.dflow INFO: Task 1175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f3e0aef379f7354f34f5e80a82e9ebb with result from task 1175
parsl.dataflow.dflow DEBUG: Task 1175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/84296262-942a-4563-a7f6-c6276cc2dc74_calibrate_947528_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/84296262-942a-4563-a7f6-c6276cc2dc74_calibrate_947528_28.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4562c28f40cc3ee1b2bfef53c922c97e with result from task 539
parsl.dataflow.dflow DEBUG: Task 539 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/702dc96a-2189-49ec-be62-6d573f82ebff_calibrate_947526_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/702dc96a-2189-49ec-be62-6d573f82ebff_calibrate_947526_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 789471b23f8550e2d989d95aedf99cc7 with result from task 994
parsl.dataflow.dflow DEBUG: Task 994 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 58f3caba6f5ab022bac70ce7dd081f0d
parsl.dataflow.memoization INFO: Task 995 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1237
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/338286b2-94b3-4005-9f04-86a7dd2c2b91_calibrate_947420_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/338286b2-94b3-4005-9f04-86a7dd2c2b91_calibrate_947420_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/dae1280d-d1e0-4ccf-9125-9aedc8b6a574_characterizeImage_947420_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/dae1280d-d1e0-4ccf-9125-9aedc8b6a574_characterizeImage_947420_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1690744ee9684ae01a7c1ac7631761f with result from task 608
parsl.dataflow.dflow DEBUG: Task 608 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7d34ade7-0908-4c73-bbfa-a6f18075fac2_calibrate_947418_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7d34ade7-0908-4c73-bbfa-a6f18075fac2_calibrate_947418_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4fb68848712440a4d0f2c2bffe78267 with result from task 629
parsl.dataflow.dflow DEBUG: Task 629 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/75b2b74a-6439-4da4-bf21-ca4f7431dc19_calibrate_947422_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/75b2b74a-6439-4da4-bf21-ca4f7431dc19_calibrate_947422_49.stderr
parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e497af5e9225edfd599bb8e4a2103679 with result from task 916
parsl.dataflow.dflow DEBUG: Task 916 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 4befb1bc7469ca4ebd3b13ef27996a1e
parsl.dataflow.memoization INFO: Task 917 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 1238
parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/907305c3-39c2-4363-b82c-8af8c6766307_calibrate_947422_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/907305c3-39c2-4363-b82c-8af8c6766307_calibrate_947422_57.stderr
parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8d9dad21-1939-4051-830f-2f553f67640c_characterizeImage_947422_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/8d9dad21-1939-4051-830f-2f553f67640c_characterizeImage_947422_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:03 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a8a11ecafe1b0301c753e00fc13639e with result from task 485
parsl.dataflow.dflow DEBUG: Task 485 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/62114cc7-f8c0-46b5-a30a-1cc4396971c4_calibrate_947528_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/62114cc7-f8c0-46b5-a30a-1cc4396971c4_calibrate_947528_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d0f1f7af635d909909fd2b807a1549d with result from task 407
parsl.dataflow.dflow DEBUG: Task 407 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e4a945b2-9a76-4120-a18a-dcdaae3d1021_calibrate_947422_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e4a945b2-9a76-4120-a18a-dcdaae3d1021_calibrate_947422_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b396b5cd02af66caaef2e7d0287521e with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0508b3ba-7a72-4d56-83c2-b9c44ff24a09_calibrate_947422_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0508b3ba-7a72-4d56-83c2-b9c44ff24a09_calibrate_947422_60.stderr
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64a57e1c93a758c68f8ebd5f13abf6ae with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 22c72a25421d478a177877b27e05b52d
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa466f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 1239
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d41f3698-1893-450d-bed3-b4c0d98a6e39_calibrate_947524_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d41f3698-1893-450d-bed3-b4c0d98a6e39_calibrate_947524_37.stderr
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a71beaf-22de-4260-903c-9fedf146a209_characterizeImage_947524_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/0a71beaf-22de-4260-903c-9fedf146a209_characterizeImage_947524_37.stderr
parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0863d3819318a0d4d6a0ec39672f377 with result from task 934
parsl.dataflow.dflow DEBUG: Task 934 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 252fa90337c2b6d8c17ee7e218efc68b
parsl.dataflow.memoization INFO: Task 935 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 1240
parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a759ce46-4b9e-493e-8747-639784d38beb_calibrate_947526_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a759ce46-4b9e-493e-8747-639784d38beb_calibrate_947526_35.stderr
parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bc3e1851-575a-4d68-8ad9-cb2136ce6804_characterizeImage_947526_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/bc3e1851-575a-4d68-8ad9-cb2136ce6804_characterizeImage_947526_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fea46b37a28392823df0afb216a23ae3 with result from task 631
parsl.dataflow.dflow DEBUG: Task 631 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 4cc05375195139bd140c244a7e832a2a
parsl.dataflow.memoization INFO: Task 632 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 1241
parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5ba3e220-22b1-4911-ac23-0fa983f40fbb_calibrate_947523_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5ba3e220-22b1-4911-ac23-0fa983f40fbb_calibrate_947523_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4c6089a6-e545-453c-b735-410e59c61d30_characterizeImage_947523_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/4c6089a6-e545-453c-b735-410e59c61d30_characterizeImage_947523_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:04 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20509b79b605ffad5bb0f3173eb06e6d with result from task 272
parsl.dataflow.dflow DEBUG: Task 272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6cc014df-c884-477a-9564-47e59fd0bab8_calibrate_947418_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/6cc014df-c884-477a-9564-47e59fd0bab8_calibrate_947418_29.stderr
parsl.dataflow.dflow INFO: Task 1126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a372ae0d660fa27d421c25cd24c4458 with result from task 1126
parsl.dataflow.dflow DEBUG: Task 1126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1127 has memoization hash 501021f75606a54bd6891ba0c0dbd784
parsl.dataflow.memoization INFO: Task 1127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1127 try 0 launched on executor multi with executor id 1242
parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e14f585c-2c87-45f1-975e-afbe7965e0fb_calibrate_947422_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e14f585c-2c87-45f1-975e-afbe7965e0fb_calibrate_947422_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/93d2eaad-4d49-45e3-9343-7009033e4be7_characterizeImage_947422_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/93d2eaad-4d49-45e3-9343-7009033e4be7_characterizeImage_947422_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0445280edf8a3af5564d69941e51adc0 with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c59a2f63-d186-4305-92d5-97bfb35e238b_calibrate_947528_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c59a2f63-d186-4305-92d5-97bfb35e238b_calibrate_947528_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:05 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8e9c9cf946c9e9139fb49157735f8396 with result from task 770
parsl.dataflow.dflow DEBUG: Task 770 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/0fd29e59-5ecb-4410-9cae-9eb59d0ba690_calibrate_947524_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/0fd29e59-5ecb-4410-9cae-9eb59d0ba690_calibrate_947524_42.stderr
parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 110b6c753d17f3f6ce79727bd2490aa5 with result from task 368
parsl.dataflow.dflow DEBUG: Task 368 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6b60cde9-acdd-4a63-ae69-5524a9f6dd3f_calibrate_947528_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6b60cde9-acdd-4a63-ae69-5524a9f6dd3f_calibrate_947528_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5642b5421fdaa4044d9aaa1f877857f7 with result from task 884
parsl.dataflow.dflow DEBUG: Task 884 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6aabdcc8-00c1-4a8d-89b9-e734dc3f10f9_calibrate_947528_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6aabdcc8-00c1-4a8d-89b9-e734dc3f10f9_calibrate_947528_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 753dacabc6ff7e2a3d5d72d10a191050 with result from task 332
parsl.dataflow.dflow DEBUG: Task 332 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/320759db-cbc3-4f26-865a-66fd3fb4a162_calibrate_947422_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/320759db-cbc3-4f26-865a-66fd3fb4a162_calibrate_947422_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 1157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1058ab0b9ee7abab1aeb84ed8f69e51e with result from task 1157
parsl.dataflow.dflow DEBUG: Task 1157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/9fd270f8-fcc2-4d01-8a4c-e6a7e6b87bb8_calibrate_947526_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 1157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/9fd270f8-fcc2-4d01-8a4c-e6a7e6b87bb8_calibrate_947526_36.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 060bf9860aa707f7d83999d645143861 with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/56bc831b-0bff-4140-a429-8ea8782c9167_calibrate_947526_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/56bc831b-0bff-4140-a429-8ea8782c9167_calibrate_947526_21.stderr
parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 034985de074405d16f0c35e380bff0b7 with result from task 1055
parsl.dataflow.dflow DEBUG: Task 1055 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f3156a5e-a4e4-4413-825a-2382eb13d7f3_calibrate_947523_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/f3156a5e-a4e4-4413-825a-2382eb13d7f3_calibrate_947523_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aae27c829ba7bd7a1ed7e11cd676922d with result from task 524
parsl.dataflow.dflow DEBUG: Task 524 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/4f9f8070-b52e-436c-95e3-4f7b515cd9f0_calibrate_947422_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/4f9f8070-b52e-436c-95e3-4f7b515cd9f0_calibrate_947422_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:06 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b56dd1643b50b37dee89f26206a50f50 with result from task 866
parsl.dataflow.dflow DEBUG: Task 866 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/035453b7-2a9a-4988-9c55-f5e6ff9ed280_calibrate_947526_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/035453b7-2a9a-4988-9c55-f5e6ff9ed280_calibrate_947526_16.stderr
parsl.dataflow.dflow INFO: Task 1172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c82ad4a0c466ab109d1b7f99c2da46f6 with result from task 1172
parsl.dataflow.dflow DEBUG: Task 1172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7cc661b7-de1a-4ba4-9290-ac715975a5b4_calibrate_947526_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7cc661b7-de1a-4ba4-9290-ac715975a5b4_calibrate_947526_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b879c10b1d44303bf0ce9c69ed29f6a5 with result from task 314
parsl.dataflow.dflow DEBUG: Task 314 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3429d20a-e0a2-49c2-b24a-6ab4d83ee35c_calibrate_947420_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3429d20a-e0a2-49c2-b24a-6ab4d83ee35c_calibrate_947420_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5268f2cf4fbfe3c928ed386bf0d66bf with result from task 731
parsl.dataflow.dflow DEBUG: Task 731 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/56d2248a-b881-4790-9412-3764fd0ebf4b_calibrate_947420_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/56d2248a-b881-4790-9412-3764fd0ebf4b_calibrate_947420_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e96682c632d9f93f7140e20b2b0aab4 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5da5ba1a-df94-413e-b1b8-b1499daaff76_calibrate_947420_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/5da5ba1a-df94-413e-b1b8-b1499daaff76_calibrate_947420_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:07 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de3c00695bfa3631da35439774432f43 with result from task 224
parsl.dataflow.dflow DEBUG: Task 224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e6887045-cfd4-4606-852f-c3867587c46d_calibrate_947422_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e6887045-cfd4-4606-852f-c3867587c46d_calibrate_947422_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9fe905ac5969b22008e893b8c2ef95 with result from task 737
parsl.dataflow.dflow DEBUG: Task 737 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/fbf02ae6-6d9a-489f-86f1-0e69fa739582_calibrate_947528_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/fbf02ae6-6d9a-489f-86f1-0e69fa739582_calibrate_947528_27.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2396a2412423cd42124eca4a9ab584fa with result from task 548
parsl.dataflow.dflow DEBUG: Task 548 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b05941a3-5b7f-4309-8cc3-973524852788_calibrate_947526_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b05941a3-5b7f-4309-8cc3-973524852788_calibrate_947526_47.stderr
parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f25881d8d2d424825d3b48f2bd1773fb with result from task 950
parsl.dataflow.dflow DEBUG: Task 950 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4548627b-a598-4bb8-9752-7301fa381524_calibrate_947420_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/4548627b-a598-4bb8-9752-7301fa381524_calibrate_947420_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc9e557174a63a8e064e41f92bec8768 with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e7036893-6303-484c-9d4a-c8255ac10c60_calibrate_947524_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e7036893-6303-484c-9d4a-c8255ac10c60_calibrate_947524_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b3914b636e1f267d2163d4aff10be67a with result from task 340
parsl.dataflow.dflow DEBUG: Task 340 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 634ba6fcf1fd23848819e8ec6c8ee6ce
parsl.dataflow.memoization INFO: Task 341 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 1243
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b9c264ad-8995-44e2-93be-d78465a5f778_calibrate_947524_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b9c264ad-8995-44e2-93be-d78465a5f778_calibrate_947524_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a2ab6cfb-2e8c-4e4b-8273-904d25d54f0c_characterizeImage_947524_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a2ab6cfb-2e8c-4e4b-8273-904d25d54f0c_characterizeImage_947524_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:08 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee3d7e5d145281263b39b384b06f9f09 with result from task 626
parsl.dataflow.dflow DEBUG: Task 626 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bf17c699-c9a8-4830-b831-8a3d1b621df5_calibrate_947420_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/bf17c699-c9a8-4830-b831-8a3d1b621df5_calibrate_947420_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16d7a580f2ab1d4cf8ab1a726c1b7744 with result from task 235
parsl.dataflow.dflow DEBUG: Task 235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 3e669b8fb2d2bb4f2474a6e25d190517
parsl.dataflow.memoization INFO: Task 236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 1244
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/182b599e-1182-47c7-b22d-585f80cc1e5d_calibrate_947526_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/182b599e-1182-47c7-b22d-585f80cc1e5d_calibrate_947526_4.stderr
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/93eaa5a6-8f48-4e06-aee5-c102880807d3_characterizeImage_947526_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/93eaa5a6-8f48-4e06-aee5-c102880807d3_characterizeImage_947526_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c429b5f14bdbd6ed0f9a0a3d2c135b6f with result from task 1294
parsl.dataflow.dflow DEBUG: Task 1294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1295 has memoization hash f667699996d7a9d4dffcce83c8b46210
parsl.dataflow.memoization INFO: Task 1295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1295 try 0 launched on executor multi with executor id 1245
parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f92218b2-8bd6-4d95-99c3-a11edcf5fcb9_calibrate_947528_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f92218b2-8bd6-4d95-99c3-a11edcf5fcb9_calibrate_947528_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e0bf357-70db-453a-9e6c-d06cf3338a6f_characterizeImage_947528_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/8e0bf357-70db-453a-9e6c-d06cf3338a6f_characterizeImage_947528_41.stderr
parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c84c89dd6ae018171e9fd4962f8c3064 with result from task 566
parsl.dataflow.dflow DEBUG: Task 566 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/508252aa-a691-41bc-8900-181305df0eb9_calibrate_947526_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/508252aa-a691-41bc-8900-181305df0eb9_calibrate_947526_59.stderr
parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c14ba264c9baedbfdb7aa6388f36facf with result from task 503
parsl.dataflow.dflow DEBUG: Task 503 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/611d1fb3-8a41-4b22-9b7d-c57c3b764fa5_calibrate_947528_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/611d1fb3-8a41-4b22-9b7d-c57c3b764fa5_calibrate_947528_1.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:09 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3f217129036e2b198a382ce8fc48c3b with result from task 967
parsl.dataflow.dflow DEBUG: Task 967 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 968 has memoization hash ec76a77c9fc58db6960943d20723df7f
parsl.dataflow.memoization INFO: Task 968 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 1246
parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/30ac6cb7-f48b-4c73-9477-0a889d37deed_calibrate_947524_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/30ac6cb7-f48b-4c73-9477-0a889d37deed_calibrate_947524_40.stderr
parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/08142465-1f37-4e4b-ae1c-5710b37b6f8f_characterizeImage_947524_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/08142465-1f37-4e4b-ae1c-5710b37b6f8f_characterizeImage_947524_40.stderr
parsl.dataflow.dflow INFO: Task 1181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60847ce4cbe10e0e9e6552102ee9af2b with result from task 1181
parsl.dataflow.dflow DEBUG: Task 1181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/caec9a05-50ab-4127-9ca0-22e1f53bceb0_calibrate_947523_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/caec9a05-50ab-4127-9ca0-22e1f53bceb0_calibrate_947523_12.stderr
parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c94770ac45dd696a4adb00b74a1cb2c with result from task 316
parsl.dataflow.dflow DEBUG: Task 316 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 317 has memoization hash c9f72c10dbba5827c074bc3699552230
parsl.dataflow.memoization INFO: Task 317 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 1247
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/017e779a-10fb-4e2b-8240-74ebbc7c4a7c_calibrate_947526_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/017e779a-10fb-4e2b-8240-74ebbc7c4a7c_calibrate_947526_6.stderr
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d27506b9-ca57-4c5e-b64c-5c49fde95470_characterizeImage_947526_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d27506b9-ca57-4c5e-b64c-5c49fde95470_characterizeImage_947526_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9399a1ab5a7031a52ab3e20b556aa62 with result from task 256
parsl.dataflow.dflow DEBUG: Task 256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 257 has memoization hash e037f9ed7dc5644c3ab5cfe07d51b2ec
parsl.dataflow.memoization INFO: Task 257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 1248
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/92cadb84-642a-474e-a712-230a0a0b83da_calibrate_947528_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/92cadb84-642a-474e-a712-230a0a0b83da_calibrate_947528_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f0a42a5b-7f27-47fd-8e96-12654c01ba68_characterizeImage_947528_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/f0a42a5b-7f27-47fd-8e96-12654c01ba68_characterizeImage_947528_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb9ae21bfec2233ba66067068d7ea276 with result from task 1300
parsl.dataflow.dflow DEBUG: Task 1300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1301 has memoization hash 4138f9a10a227b480f4ad71e655cf8e7
parsl.dataflow.memoization INFO: Task 1301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1301 try 0 launched on executor multi with executor id 1249
parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ad06130a-7d6d-45ce-8afe-172d87dda79c_calibrate_947418_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ad06130a-7d6d-45ce-8afe-172d87dda79c_calibrate_947418_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cf0725c3-d177-47a8-833c-1f3c7723a383_characterizeImage_947418_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/cf0725c3-d177-47a8-833c-1f3c7723a383_characterizeImage_947418_38.stderr
parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38963556736b31d54dbe42fb348a306a with result from task 602
parsl.dataflow.dflow DEBUG: Task 602 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f07fef78-0437-4016-a574-31e1ec3bf024_calibrate_947526_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f07fef78-0437-4016-a574-31e1ec3bf024_calibrate_947526_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b6e10a99fed9990255e8a0002dd6197 with result from task 479
parsl.dataflow.dflow DEBUG: Task 479 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2a58143b-6e1a-429d-933e-8a2cdfd2fd7d_calibrate_947528_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2a58143b-6e1a-429d-933e-8a2cdfd2fd7d_calibrate_947528_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f602c3aab909837dd0e8ad4699279b90 with result from task 896
parsl.dataflow.dflow DEBUG: Task 896 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b7196490-e793-4b90-87fa-067f1bc08b3b_calibrate_947528_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/b7196490-e793-4b90-87fa-067f1bc08b3b_calibrate_947528_4.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:10 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 455b15e0cfc1c345095b7f3d294c9418 with result from task 1106
parsl.dataflow.dflow DEBUG: Task 1106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f0b19d00-91f9-438f-9e60-72d5b243cee3_calibrate_947422_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/f0b19d00-91f9-438f-9e60-72d5b243cee3_calibrate_947422_43.stderr
parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa7fc1c399290ab60fc18603cd487de with result from task 989
parsl.dataflow.dflow DEBUG: Task 989 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ba97eb2d-d4ce-40f9-a913-9e17a60b78ae_calibrate_947420_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ba97eb2d-d4ce-40f9-a913-9e17a60b78ae_calibrate_947420_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2de4ab23a83be1bc77cd8fcf60cd3b4d with result from task 335
parsl.dataflow.dflow DEBUG: Task 335 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/92cdb360-9aeb-4a72-b12a-7d8876145a7c_calibrate_947524_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/92cdb360-9aeb-4a72-b12a-7d8876145a7c_calibrate_947524_34.stderr
parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c752acec1b753c9f29431ba248c9a5c with result from task 734
parsl.dataflow.dflow DEBUG: Task 734 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/bc4e2906-0428-4a58-9aa3-6d822eb58837_calibrate_947524_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/bc4e2906-0428-4a58-9aa3-6d822eb58837_calibrate_947524_43.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0bbb1893f4a6916f67f260d8dfe43549 with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a6138eac-9f4a-4ac4-afcc-a1fcf608d04c_calibrate_947422_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a6138eac-9f4a-4ac4-afcc-a1fcf608d04c_calibrate_947422_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1baee34f9b75c7be236cf38d515fb28b with result from task 620
parsl.dataflow.dflow DEBUG: Task 620 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0792b4e1-2dc1-45e1-ab04-f36921d2c00f_calibrate_947526_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0792b4e1-2dc1-45e1-ab04-f36921d2c00f_calibrate_947526_60.stderr
parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d39bf69ddac91056d1e1921961a7db26 with result from task 460
parsl.dataflow.dflow DEBUG: Task 460 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 8bf6c75fb1ab803e93f1750464ae1bad
parsl.dataflow.memoization INFO: Task 461 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d84860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 1250
parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/c5290151-0427-4946-9278-5dc73c454987_calibrate_947526_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/c5290151-0427-4946-9278-5dc73c454987_calibrate_947526_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2637e1be-a477-4e5f-b747-8317cbc909af_characterizeImage_947526_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/2637e1be-a477-4e5f-b747-8317cbc909af_characterizeImage_947526_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52d16b34deb771599acf23300f1e0927 with result from task 572
parsl.dataflow.dflow DEBUG: Task 572 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ca0a7610-86d3-4891-bd87-a2183f83628d_calibrate_947420_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/ca0a7610-86d3-4891-bd87-a2183f83628d_calibrate_947420_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:11 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb466ac17f813fc0c1461ebb5a2e22e3 with result from task 953
parsl.dataflow.dflow DEBUG: Task 953 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a1da334-4ce9-4ee2-8340-6514909d15a0_calibrate_947523_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a1da334-4ce9-4ee2-8340-6514909d15a0_calibrate_947523_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 462b0d7d6ce20e7635d7e9205b3cb8f8 with result from task 560
parsl.dataflow.dflow DEBUG: Task 560 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5a063e76-ed01-4a14-802a-80f83a3c0ca2_calibrate_947528_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5a063e76-ed01-4a14-802a-80f83a3c0ca2_calibrate_947528_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:12 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ddeb76f042b8119639e1305dee0f798 with result from task 1034
parsl.dataflow.dflow DEBUG: Task 1034 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/63920ee9-68e7-4132-ac1a-4eef95756baa_calibrate_947422_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/63920ee9-68e7-4132-ac1a-4eef95756baa_calibrate_947422_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 64eb349dd9c93a22ef7083e2f9c2844e with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 00d90054c9dac77c60b575a2ccdee1fb
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 1251
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e69c6151-35fc-4286-99e0-c0234aa5f17f_calibrate_947528_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e69c6151-35fc-4286-99e0-c0234aa5f17f_calibrate_947528_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ba42188d-afdb-44d6-9a21-5a59fd186fd1_characterizeImage_947528_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/ba42188d-afdb-44d6-9a21-5a59fd186fd1_characterizeImage_947528_61.stderr
parsl.dataflow.dflow INFO: Task 1219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8eaaf68106f2988a70fc862659120d10 with result from task 1219
parsl.dataflow.dflow DEBUG: Task 1219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1220 has memoization hash 1b4766f90f1be7cf66f674896d559880
parsl.dataflow.memoization INFO: Task 1220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1220 try 0 launched on executor multi with executor id 1252
parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/93e53ec7-34df-45bd-8332-a100a15e53cf_calibrate_947526_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/93e53ec7-34df-45bd-8332-a100a15e53cf_calibrate_947526_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ff5aa620-879f-4f13-b469-3007508d0cad_characterizeImage_947526_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/ff5aa620-879f-4f13-b469-3007508d0cad_characterizeImage_947526_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d71444607b4604a4038d601dd5a87e82 with result from task 1259
parsl.dataflow.dflow DEBUG: Task 1259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c7571885-1a22-4723-b3cc-58c1122bdf80_calibrate_947422_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 1259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c7571885-1a22-4723-b3cc-58c1122bdf80_calibrate_947422_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84b4befb410f729cf08072fb11603a96 with result from task 854
parsl.dataflow.dflow DEBUG: Task 854 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/562cd549-a53a-4d1d-bd2b-73d9c199cba7_calibrate_947524_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/562cd549-a53a-4d1d-bd2b-73d9c199cba7_calibrate_947524_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 088bf45f8d4cc4bcf3f5b988b3010180 with result from task 767
parsl.dataflow.dflow DEBUG: Task 767 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5cf5ae4e-36f3-4897-aee7-1db344dc269e_calibrate_947528_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5cf5ae4e-36f3-4897-aee7-1db344dc269e_calibrate_947528_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:13 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ac99585b8e65d182271359ff83f7771 with result from task 1031
parsl.dataflow.dflow DEBUG: Task 1031 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cccdd7b6-19b1-4563-8bb6-2a6b0ec82fe9_calibrate_947528_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cccdd7b6-19b1-4563-8bb6-2a6b0ec82fe9_calibrate_947528_13.stderr
parsl.dataflow.dflow INFO: Task 1225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae735ec6d303db0f29038cfa71b38aed with result from task 1225
parsl.dataflow.dflow DEBUG: Task 1225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1226 has memoization hash 44525d9425fb854f4cc672e0a2faf434
parsl.dataflow.memoization INFO: Task 1226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1226 try 0 launched on executor multi with executor id 1253
parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f2c07152-6a29-4a15-8f51-43a10209ff3e_calibrate_947524_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f2c07152-6a29-4a15-8f51-43a10209ff3e_calibrate_947524_21.stderr
parsl.dataflow.dflow INFO: Standard output for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9375c84c-85df-42bf-aa8c-5de7b93d0b6e_characterizeImage_947524_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/9375c84c-85df-42bf-aa8c-5de7b93d0b6e_characterizeImage_947524_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 855c4ec8333e109da7bdb61ca28a8bf1 with result from task 845
parsl.dataflow.dflow DEBUG: Task 845 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/377ceece-851a-4faf-8172-99103d7fb040_calibrate_947523_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/377ceece-851a-4faf-8172-99103d7fb040_calibrate_947523_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44fe01d8882a3e7838f314f538ad5b48 with result from task 1261
parsl.dataflow.dflow DEBUG: Task 1261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1262 has memoization hash 3b911d1d95eebd6a2013ad7cb3221d43
parsl.dataflow.memoization INFO: Task 1262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1262 try 0 launched on executor multi with executor id 1254
parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7d149435-895b-428e-a8cf-9685cd943f58_calibrate_947524_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7d149435-895b-428e-a8cf-9685cd943f58_calibrate_947524_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c93cbd81-ada3-4c5e-b47e-0488bd33f09b_characterizeImage_947524_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/c93cbd81-ada3-4c5e-b47e-0488bd33f09b_characterizeImage_947524_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b272cdb0766459285ae61e835457ec9 with result from task 827
parsl.dataflow.dflow DEBUG: Task 827 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/195c6bb2-b831-451b-b169-3e52d6a56dc2_calibrate_947528_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/195c6bb2-b831-451b-b169-3e52d6a56dc2_calibrate_947528_56.stderr
parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d6ce0cc7422d21eded29b38add7c659b with result from task 377
parsl.dataflow.dflow DEBUG: Task 377 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/65b7d837-ca27-4629-988a-fc8d46887d5c_calibrate_947524_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/65b7d837-ca27-4629-988a-fc8d46887d5c_calibrate_947524_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9604a776e533cf26dce9b127ff0dfc22 with result from task 259
parsl.dataflow.dflow DEBUG: Task 259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 93aab17e34b12055f9689edb5357a073
parsl.dataflow.memoization INFO: Task 260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 1255
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a408b5cf-0940-4a5f-9d04-130fd3e53855_calibrate_947528_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a408b5cf-0940-4a5f-9d04-130fd3e53855_calibrate_947528_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/61d102c0-2d53-4d7e-b401-94c550e6b1c5_characterizeImage_947528_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/61d102c0-2d53-4d7e-b401-94c550e6b1c5_characterizeImage_947528_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:14 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:15 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 59978bc752e0b9581445b4c4fae9e387 with result from task 1022
parsl.dataflow.dflow DEBUG: Task 1022 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7346fe4d-dc7d-436d-95b6-e1b30eb45bb9_calibrate_947418_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/7346fe4d-dc7d-436d-95b6-e1b30eb45bb9_calibrate_947418_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:15 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d77f6e687fc01db8a8ca7a6e9ebc8353 with result from task 922
parsl.dataflow.dflow DEBUG: Task 922 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 923 has memoization hash ba02893359489af24a4e6e6c060a274b
parsl.dataflow.memoization INFO: Task 923 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 1256
parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/06af0cd0-b7fb-4187-9426-2f82a7ab3c3a_calibrate_947526_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/06af0cd0-b7fb-4187-9426-2f82a7ab3c3a_calibrate_947526_51.stderr
parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/38492558-d307-45ac-a7c0-95537f9b8fc9_characterizeImage_947526_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/38492558-d307-45ac-a7c0-95537f9b8fc9_characterizeImage_947526_51.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:15 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:15 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e1ffb4849fdf53ad5508c4295f41dad with result from task 1147
parsl.dataflow.dflow DEBUG: Task 1147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1148 has memoization hash 4b9d90aeb305d995556e5bd135048db6
parsl.dataflow.memoization INFO: Task 1148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1148 try 0 launched on executor multi with executor id 1257
parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7e43ca49-e3ff-4ff7-aede-ce6810c57a25_calibrate_947524_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7e43ca49-e3ff-4ff7-aede-ce6810c57a25_calibrate_947524_9.stderr
parsl.dataflow.dflow INFO: Standard output for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ea2c01ab-4303-4896-9489-4d9980fbf245_characterizeImage_947524_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/ea2c01ab-4303-4896-9489-4d9980fbf245_characterizeImage_947524_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:15 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 171
parsl.dataflow.strategy DEBUG: Executor multi has 171 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 171
parsl.dataflow.strategy DEBUG: Executor multi has 171 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0732022401410620dc72e4f27c60f872 with result from task 860
parsl.dataflow.dflow DEBUG: Task 860 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/736eb83a-b81a-4a2d-b86d-28dc6933b0cd_calibrate_947422_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/736eb83a-b81a-4a2d-b86d-28dc6933b0cd_calibrate_947422_51.stderr
parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4905fad1fcb48ea5721eef633a797004 with result from task 965
parsl.dataflow.dflow DEBUG: Task 965 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/848aff7f-9b50-4007-b7e0-e2bd3e057bc3_calibrate_947523_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/848aff7f-9b50-4007-b7e0-e2bd3e057bc3_calibrate_947523_35.stderr
parsl.dataflow.dflow INFO: Task 1282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3a86a0213e9d40965977f48285cc1a0 with result from task 1282
parsl.dataflow.dflow DEBUG: Task 1282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1283 has memoization hash c0f04391dd4ebc96aba0cc2ed8d1fcf5
parsl.dataflow.memoization INFO: Task 1283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1283 try 0 launched on executor multi with executor id 1258
parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/39217143-35e1-4a8c-8a95-c8d3f1251b74_calibrate_947523_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/39217143-35e1-4a8c-8a95-c8d3f1251b74_calibrate_947523_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/927ef615-cb14-4848-a1c8-f01c63223f31_characterizeImage_947523_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/927ef615-cb14-4848-a1c8-f01c63223f31_characterizeImage_947523_25.stderr
parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c43279ffb5e28aa21467ef56ca25a80 with result from task 1073
parsl.dataflow.dflow DEBUG: Task 1073 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f16312da-0ef1-4432-b252-349e9f9c2821_calibrate_947524_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f16312da-0ef1-4432-b252-349e9f9c2821_calibrate_947524_22.stderr
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6954da9d490a1be32eea10181b8ea0d with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 152a1ebb6acc7a9d0e7be0b2b995dfb8
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6fa4a49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 1259
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ba50c37f-b222-49c0-a915-e300741a015c_calibrate_947524_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ba50c37f-b222-49c0-a915-e300741a015c_calibrate_947524_60.stderr
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/685a7bbe-0e9f-4da2-b106-1acc0c445408_characterizeImage_947524_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/685a7bbe-0e9f-4da2-b106-1acc0c445408_characterizeImage_947524_60.stderr
parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15f3abf2b01794f68b15f624505855da with result from task 890
parsl.dataflow.dflow DEBUG: Task 890 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d1f00a36-3b47-4441-bc8b-0f97aed5531c_calibrate_947526_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d1f00a36-3b47-4441-bc8b-0f97aed5531c_calibrate_947526_33.stderr
parsl.dataflow.dflow INFO: Task 1205 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 548e2d352438db7b8d041e0e2169fb31 with result from task 1205
parsl.dataflow.dflow DEBUG: Task 1205 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3786ea10-43dc-4f3f-b4c1-3a5d24b4a1a0_calibrate_947420_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/3786ea10-43dc-4f3f-b4c1-3a5d24b4a1a0_calibrate_947420_37.stderr
parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 39fbb77024ba043d27e67537bc9b8620 with result from task 506
parsl.dataflow.dflow DEBUG: Task 506 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/33caab7d-a1fe-4e84-b563-a0021c0ee47f_calibrate_947524_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/33caab7d-a1fe-4e84-b563-a0021c0ee47f_calibrate_947524_13.stderr
parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96fd4717f4b49fce4b8ce7ee6505a93f with result from task 365
parsl.dataflow.dflow DEBUG: Task 365 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/19c6dd54-14aa-4316-841e-f54f46fd49f8_calibrate_947420_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/19c6dd54-14aa-4316-841e-f54f46fd49f8_calibrate_947420_30.stderr
parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a721d2640ac47ade5644b6d22f729ad6 with result from task 1097
parsl.dataflow.dflow DEBUG: Task 1097 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a891c0df-30ef-4902-b618-34bf467e87bf_calibrate_947524_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a891c0df-30ef-4902-b618-34bf467e87bf_calibrate_947524_12.stderr
parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ffb081c7588a367c0f48ebe34a504705 with result from task 1016
parsl.dataflow.dflow DEBUG: Task 1016 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/077104cc-8ffc-4820-aa15-140b64ecfc78_calibrate_947420_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/077104cc-8ffc-4820-aa15-140b64ecfc78_calibrate_947420_40.stderr
parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f4347734a3f503f453d5bb514f89534 with result from task 1019
parsl.dataflow.dflow DEBUG: Task 1019 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/73741d9f-58af-4110-b126-42107d5fda32_calibrate_947420_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/73741d9f-58af-4110-b126-42107d5fda32_calibrate_947420_5.stderr
parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9df2f3a4a1c84ef1eea1bbe48f94336a with result from task 296
parsl.dataflow.dflow DEBUG: Task 296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f90c8c0-54b7-495f-b702-3eb1ef40d25a_calibrate_947523_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6f90c8c0-54b7-495f-b702-3eb1ef40d25a_calibrate_947523_19.stderr
parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6552b0f9955d7b9004d4da17545c9a7d with result from task 1039
parsl.dataflow.dflow DEBUG: Task 1039 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 27545f2fc607e58111b2678ed7fea055
parsl.dataflow.memoization INFO: Task 1040 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 1260
parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/363943fd-9d46-4af0-b2cc-887d2f62351f_calibrate_947526_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/363943fd-9d46-4af0-b2cc-887d2f62351f_calibrate_947526_41.stderr
parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/976861c6-3757-438c-89d0-92dc16adcbd6_characterizeImage_947526_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/976861c6-3757-438c-89d0-92dc16adcbd6_characterizeImage_947526_41.stderr
parsl.dataflow.dflow INFO: Task 1241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c142547641a580b44ba182eabfea0009 with result from task 1241
parsl.dataflow.dflow DEBUG: Task 1241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c804e15c-24a6-43ea-829a-80b3a4e9442e_calibrate_947523_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/c804e15c-24a6-43ea-829a-80b3a4e9442e_calibrate_947523_3.stderr
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1b8168a8e54d6b92317b3792f3c9a7c with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/19d291d3-0c38-41fb-80d6-90119f0790bb_calibrate_947524_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/19d291d3-0c38-41fb-80d6-90119f0790bb_calibrate_947524_54.stderr
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6728c2cf94c7c7e9c771e30b3d841dcd with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c1378830-4e1a-4fe4-a07e-6e6e1f65dd44_calibrate_947422_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/c1378830-4e1a-4fe4-a07e-6e6e1f65dd44_calibrate_947422_40.stderr
parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b04601b9a4f19b29b7d76a80e1019575 with result from task 292
parsl.dataflow.dflow DEBUG: Task 292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 8a29bd8b6494b94ff0202b5d9dc0991e
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 1261
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f87bb675-adbd-453a-8d3d-c6a4f1f75825_calibrate_947524_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f87bb675-adbd-453a-8d3d-c6a4f1f75825_calibrate_947524_8.stderr
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7071bda8-6073-47aa-9d52-7bd022c68cbe_characterizeImage_947524_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7071bda8-6073-47aa-9d52-7bd022c68cbe_characterizeImage_947524_8.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd529a776accb4fc52155509f2a41040 with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/237f2531-ae62-428b-b29d-18abc6068afa_calibrate_947528_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/237f2531-ae62-428b-b29d-18abc6068afa_calibrate_947528_6.stderr
parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea8fd94e29986e7f1ac5b9a1e29aaa3c with result from task 841
parsl.dataflow.dflow DEBUG: Task 841 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 3ce3e6645a807a73da36bb2cb694d99c
parsl.dataflow.memoization INFO: Task 842 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d098a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 1262
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d81daa5a-4713-487a-856a-88bf9141b86c_characterizeImage_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/d81daa5a-4713-487a-856a-88bf9141b86c_characterizeImage_947526_61.stderr
parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb9cfe2702316702b82bcd56b9f47f5b with result from task 920
parsl.dataflow.dflow DEBUG: Task 920 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8916b295-0512-4031-9840-59b28de38aae_calibrate_947422_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8916b295-0512-4031-9840-59b28de38aae_calibrate_947422_20.stderr
parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9227d04482233a40557dea419d9aeb0a with result from task 445
parsl.dataflow.dflow DEBUG: Task 445 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 7a2c8113ccf8b90c7ee749f3674b251c
parsl.dataflow.memoization INFO: Task 446 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f53880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 1263
parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/2e8a6bc9-56f4-4ff8-9113-850715aeb0ff_calibrate_947422_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/2e8a6bc9-56f4-4ff8-9113-850715aeb0ff_calibrate_947422_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7f8f9ddd-c04f-4403-8367-fa1bf33c85c1_characterizeImage_947422_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/7f8f9ddd-c04f-4403-8367-fa1bf33c85c1_characterizeImage_947422_46.stderr
parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4d73ff6fe88c044759ccd261578d7fd with result from task 1058
parsl.dataflow.dflow DEBUG: Task 1058 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/55054fed-ffde-488b-b4ae-fc7e8aa815e5_calibrate_947528_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/55054fed-ffde-488b-b4ae-fc7e8aa815e5_calibrate_947528_23.stderr
parsl.dataflow.dflow INFO: Task 1250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc8045e77388a98f4a4edfd0ba2776c with result from task 1250
parsl.dataflow.dflow DEBUG: Task 1250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/9ea592f6-d5d5-441d-bdf0-5891036ece51_calibrate_947418_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 1250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/9ea592f6-d5d5-441d-bdf0-5891036ece51_calibrate_947418_37.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 153
parsl.dataflow.strategy DEBUG: Executor multi has 153 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e92bdb5646fa4ed2fb6250072d7a4a9 with result from task 500
parsl.dataflow.dflow DEBUG: Task 500 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6fd0a284-2ef1-492b-883b-5da46a1ddf13_calibrate_947523_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/6fd0a284-2ef1-492b-883b-5da46a1ddf13_calibrate_947523_50.stderr
parsl.dataflow.dflow INFO: Task 1169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8090a3b380c5b5a6714ec4af6137418d with result from task 1169
parsl.dataflow.dflow DEBUG: Task 1169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cde31089-c869-47d9-b075-a6f4b0a3678d_calibrate_947528_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/cde31089-c869-47d9-b075-a6f4b0a3678d_calibrate_947528_35.stderr
parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ab38dc7c617c123a37fa9277674bbbd with result from task 1025
parsl.dataflow.dflow DEBUG: Task 1025 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e2fc2082-07a3-42dc-b894-2f9974fca693_calibrate_947418_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e2fc2082-07a3-42dc-b894-2f9974fca693_calibrate_947418_20.stderr
parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63fef84ba207294447588ad6677cde07 with result from task 527
parsl.dataflow.dflow DEBUG: Task 527 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/44343698-783d-401a-bdb6-0343ed9f6e2f_calibrate_947418_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/44343698-783d-401a-bdb6-0343ed9f6e2f_calibrate_947418_17.stderr
parsl.dataflow.dflow INFO: Task 1289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e94d6aaa40bb5c6e7682cc922e91a04c with result from task 1289
parsl.dataflow.dflow DEBUG: Task 1289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/cd95c9a8-521b-4ae1-8202-48947ee18636_calibrate_947523_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 1289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/cd95c9a8-521b-4ae1-8202-48947ee18636_calibrate_947523_14.stderr
parsl.dataflow.dflow INFO: Task 1255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9311ca2e86f1461f0d30caa3f269d749 with result from task 1255
parsl.dataflow.dflow DEBUG: Task 1255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1256 has memoization hash 51afcbb1d14f112aad5367e598b3d40b
parsl.dataflow.memoization INFO: Task 1256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1256 try 0 launched on executor multi with executor id 1264
parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e28f0e71-969f-4e16-b083-eb9593f4ce98_calibrate_947524_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e28f0e71-969f-4e16-b083-eb9593f4ce98_calibrate_947524_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6a0fe967-5da7-43be-827b-7d9e50e12f0c_characterizeImage_947524_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/6a0fe967-5da7-43be-827b-7d9e50e12f0c_characterizeImage_947524_39.stderr
parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4a66dd08bdb38983fcb92e5cf5cb6c0 with result from task 983
parsl.dataflow.dflow DEBUG: Task 983 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d74bddb5-e2fc-4b9c-b108-30c952796a9f_calibrate_947420_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/d74bddb5-e2fc-4b9c-b108-30c952796a9f_calibrate_947420_29.stderr
parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb928f56a20baa0bbecebd0c7cf2c2ac with result from task 929
parsl.dataflow.dflow DEBUG: Task 929 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e549ef4d-f234-4cda-a56b-6f7d537a473a_calibrate_947523_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e549ef4d-f234-4cda-a56b-6f7d537a473a_calibrate_947523_56.stderr
parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0cb44c520d7eef493581d34d6427a36 with result from task 356
parsl.dataflow.dflow DEBUG: Task 356 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/87ce2a96-bdb4-4702-979b-b0e1f1601548_calibrate_947526_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/87ce2a96-bdb4-4702-979b-b0e1f1601548_calibrate_947526_54.stderr
parsl.dataflow.dflow INFO: Task 1223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25aaff1a3a9c72a113c52dea33d06aa9 with result from task 1223
parsl.dataflow.dflow DEBUG: Task 1223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/aa9266fc-f3c8-40f7-84ff-6dd0dd3f3088_calibrate_947523_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/aa9266fc-f3c8-40f7-84ff-6dd0dd3f3088_calibrate_947523_34.stderr
parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 564b4ad8c33f40a7601327757fb84814 with result from task 1091
parsl.dataflow.dflow DEBUG: Task 1091 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dd845476-aa5d-491c-b7f3-d1105e4561e9_calibrate_947524_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/dd845476-aa5d-491c-b7f3-d1105e4561e9_calibrate_947524_56.stderr
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9711be7c769544d39207620781f0bbed with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/69e9aa9a-5735-4614-8e22-8db95ca1bd74_calibrate_947524_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/69e9aa9a-5735-4614-8e22-8db95ca1bd74_calibrate_947524_24.stderr
parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5464f95d50d8c67c023f0601a9c929b2 with result from task 875
parsl.dataflow.dflow DEBUG: Task 875 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6ca1c8bc-9e06-4c5f-9072-46243fe01fbc_calibrate_947528_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/6ca1c8bc-9e06-4c5f-9072-46243fe01fbc_calibrate_947528_11.stderr
parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5ba59d66af99c89430cdd3a25ecd4ca with result from task 343
parsl.dataflow.dflow DEBUG: Task 343 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 344 has memoization hash bb4cfedc0d0755865005865c6f2ebb49
parsl.dataflow.memoization INFO: Task 344 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f20e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 1265
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e8d9209b-da58-4040-a427-d3556b64b616_calibrate_947418_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e8d9209b-da58-4040-a427-d3556b64b616_calibrate_947418_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/55c08191-dc62-4879-90c4-ed0d9042643e_characterizeImage_947418_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/55c08191-dc62-4879-90c4-ed0d9042643e_characterizeImage_947418_25.stderr
parsl.dataflow.dflow INFO: Task 1247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a650f6e796f7d3bb3cd18ea1757ccaf with result from task 1247
parsl.dataflow.dflow DEBUG: Task 1247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/895238d0-000a-42a7-9cd2-e39c5177c1ea_calibrate_947418_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 1247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/895238d0-000a-42a7-9cd2-e39c5177c1ea_calibrate_947418_28.stderr
parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80654a7031c2d64322bb1d6c0956d313 with result from task 425
parsl.dataflow.dflow DEBUG: Task 425 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/da32157e-4d58-4496-91ce-9bd627546ab8_calibrate_947422_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/da32157e-4d58-4496-91ce-9bd627546ab8_calibrate_947422_45.stderr
parsl.dataflow.dflow INFO: Task 1196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c79a91adde1a323b5e900f533b14700f with result from task 1196
parsl.dataflow.dflow DEBUG: Task 1196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/368771bf-d69e-433a-9b88-b23f197c4323_calibrate_947523_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 1196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/368771bf-d69e-433a-9b88-b23f197c4323_calibrate_947523_54.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1192 try 0 failed
parsl.dataflow.dflow INFO: Task 1192 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1192 has memoization hash 7c3583346baef57b8b83ea4e4b88958a
parsl.dataflow.memoization INFO: Task 1192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1192 try 1 launched on executor multi with executor id 1266
parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stderr
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b8f23aba2f763f347cacb3f4a7c465b with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e3e91131-651d-4d45-9b4a-d77107f709f8_calibrate_947523_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e3e91131-651d-4d45-9b4a-d77107f709f8_calibrate_947523_16.stderr
parsl.dataflow.dflow INFO: Task 1231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f0b714d92cdc6ba8cbe9881e8a5f946 with result from task 1231
parsl.dataflow.dflow DEBUG: Task 1231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1232 has memoization hash cc130e6b25cb0b177d4714b45b00ba8c
parsl.dataflow.memoization INFO: Task 1232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1232 try 0 launched on executor multi with executor id 1267
parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/12dc46e1-08d7-4f2f-9514-f245557f77ab_calibrate_947528_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/12dc46e1-08d7-4f2f-9514-f245557f77ab_calibrate_947528_46.stderr
parsl.dataflow.dflow INFO: Standard output for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a1558d39-b089-4ee8-85f7-2728c83433c0_characterizeImage_947528_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/a1558d39-b089-4ee8-85f7-2728c83433c0_characterizeImage_947528_46.stderr
parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c71243149308846fff4bbbe5001b4404 with result from task 557
parsl.dataflow.dflow DEBUG: Task 557 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/542e1ac6-8ea7-41aa-bc28-f32e93348243_calibrate_947418_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/542e1ac6-8ea7-41aa-bc28-f32e93348243_calibrate_947418_5.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 136
parsl.dataflow.strategy DEBUG: Executor multi has 136 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 681db473c8fe851acdbe22c49bee069c with result from task 275
parsl.dataflow.dflow DEBUG: Task 275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/206c951a-c85e-4980-bb7d-eaa4feab28d1_calibrate_947524_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/206c951a-c85e-4980-bb7d-eaa4feab28d1_calibrate_947524_53.stderr
parsl.dataflow.dflow INFO: Task 1280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26ad97da7ff91a424ad928300ad66e0b with result from task 1280
parsl.dataflow.dflow DEBUG: Task 1280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15afa65a-7887-45d8-a61d-f736de01eb02_calibrate_947420_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 1280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/15afa65a-7887-45d8-a61d-f736de01eb02_calibrate_947420_44.stderr
parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8aba59ec035e3c7dee5f9ad4d977b7d0 with result from task 299
parsl.dataflow.dflow DEBUG: Task 299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/49f6e3a1-47a3-4d18-9aad-cc954243b823_calibrate_947526_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/49f6e3a1-47a3-4d18-9aad-cc954243b823_calibrate_947526_55.stderr
parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eef86ce413aaa29579dc4bd0af106e28 with result from task 1085
parsl.dataflow.dflow DEBUG: Task 1085 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c0196206-22c3-477f-b03b-ba5686168b0e_calibrate_947528_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c0196206-22c3-477f-b03b-ba5686168b0e_calibrate_947528_24.stderr
parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9055fffe59dc310403c4cd97d958be36 with result from task 947
parsl.dataflow.dflow DEBUG: Task 947 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/275490cd-d264-411b-aa69-b15999710e9a_calibrate_947418_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/275490cd-d264-411b-aa69-b15999710e9a_calibrate_947418_8.stderr
parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac32e0e59d9d89078a6616cef4d9473 with result from task 938
parsl.dataflow.dflow DEBUG: Task 938 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/196bdee1-4400-4596-a291-ea369e596d87_calibrate_947524_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/196bdee1-4400-4596-a291-ea369e596d87_calibrate_947524_26.stderr
parsl.dataflow.dflow INFO: Task 1136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7d66b1c17ad8459f147a9615f33ede3 with result from task 1136
parsl.dataflow.dflow DEBUG: Task 1136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0fe78b18-69c8-4c01-b946-c7327a641796_calibrate_947422_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0fe78b18-69c8-4c01-b946-c7327a641796_calibrate_947422_18.stderr
parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b6e059e272b7b1abd44fb1bc77d7de2 with result from task 971
parsl.dataflow.dflow DEBUG: Task 971 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a1c3e144-f190-4659-a765-642736fd2768_calibrate_947422_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/a1c3e144-f190-4659-a765-642736fd2768_calibrate_947422_34.stderr
parsl.dataflow.dflow INFO: Task 1118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f28ccc3436eb93f194fe77094ad5de0 with result from task 1118
parsl.dataflow.dflow DEBUG: Task 1118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0501a906-bc15-410b-bd1b-28c882ce2057_calibrate_947422_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0501a906-bc15-410b-bd1b-28c882ce2057_calibrate_947422_9.stderr
parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2fa18a4609706c98c781d74d9f779fbe with result from task 803
parsl.dataflow.dflow DEBUG: Task 803 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/573a53ab-cd47-47c5-84ca-bd04ef2edf58_calibrate_947524_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/573a53ab-cd47-47c5-84ca-bd04ef2edf58_calibrate_947524_1.stderr
parsl.dataflow.dflow INFO: Task 1273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d555d1d15b9d512f470e0c7ddd6862d with result from task 1273
parsl.dataflow.dflow DEBUG: Task 1273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1274 has memoization hash f2c3e9ce99782b0a9345ed582cebed4d
parsl.dataflow.memoization INFO: Task 1274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1274 try 0 launched on executor multi with executor id 1268
parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/86c99b0a-8a37-4b13-9d77-904e0418bdb7_calibrate_947528_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/86c99b0a-8a37-4b13-9d77-904e0418bdb7_calibrate_947528_5.stderr
parsl.dataflow.dflow INFO: Standard output for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/39bfe967-4a53-4a79-8ad9-49935df3b8a6_characterizeImage_947528_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/39bfe967-4a53-4a79-8ad9-49935df3b8a6_characterizeImage_947528_5.stderr
parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6e81464f824e86d0feaf9aa48870218 with result from task 848
parsl.dataflow.dflow DEBUG: Task 848 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9e1792f7-3284-4ae2-a382-742170b50fd1_calibrate_947524_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/9e1792f7-3284-4ae2-a382-742170b50fd1_calibrate_947524_4.stderr
parsl.dataflow.dflow INFO: Task 1190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8b4dd52814a45fb518c32b57d577dd4 with result from task 1190
parsl.dataflow.dflow DEBUG: Task 1190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/7f528d70-9737-43b2-9527-dfea17eeca70_calibrate_947422_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/7f528d70-9737-43b2-9527-dfea17eeca70_calibrate_947422_7.stderr
parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7ca281cfff6cc901b900bd5e3500d92 with result from task 515
parsl.dataflow.dflow DEBUG: Task 515 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5360db-cf1a-4e0e-b4a5-cc6c48ba5c2b_calibrate_947526_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5360db-cf1a-4e0e-b4a5-cc6c48ba5c2b_calibrate_947526_49.stderr
parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70275317fac18b96b63f5d1bbaa173b2 with result from task 1061
parsl.dataflow.dflow DEBUG: Task 1061 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/431e9228-2c4b-438d-aa25-ddb70972d061_calibrate_947526_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/431e9228-2c4b-438d-aa25-ddb70972d061_calibrate_947526_39.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 122
parsl.dataflow.strategy DEBUG: Executor multi has 122 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 1129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7e9021282d5883d3288ddb659455f29 with result from task 1129
parsl.dataflow.dflow DEBUG: Task 1129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1130 has memoization hash d6659e0442146e07c1ef8010fd5ec8db
parsl.dataflow.memoization INFO: Task 1130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1130 try 0 launched on executor multi with executor id 1269
parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f2354ac5-6371-4ab7-b540-adc0dddf65af_calibrate_947526_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f2354ac5-6371-4ab7-b540-adc0dddf65af_calibrate_947526_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/128e275c-f516-4cb1-aff3-4b6951aa1dff_characterizeImage_947526_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/128e275c-f516-4cb1-aff3-4b6951aa1dff_characterizeImage_947526_58.stderr
parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e41bce103f44902a4bce7b190bd2381 with result from task 1046
parsl.dataflow.dflow DEBUG: Task 1046 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8c070fcb-74bc-4984-b9af-105c15defad6_calibrate_947422_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/8c070fcb-74bc-4984-b9af-105c15defad6_calibrate_947422_6.stderr
parsl.dataflow.dflow INFO: Task 1210 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2057d7a625dbfd7d89c6cab00a14bde0 with result from task 1210
parsl.dataflow.dflow DEBUG: Task 1210 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1211 has memoization hash 3b78edc394893ef5d3ee0d29817256a7
parsl.dataflow.memoization INFO: Task 1211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1211 try 0 launched on executor multi with executor id 1270
parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/544db9e2-862d-4282-9b2b-ca9e9b5fba25_calibrate_947526_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/544db9e2-862d-4282-9b2b-ca9e9b5fba25_calibrate_947526_29.stderr
parsl.dataflow.dflow INFO: Standard output for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f4d39cf1-ea41-4beb-b6be-14a6271560ce_characterizeImage_947526_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/f4d39cf1-ea41-4beb-b6be-14a6271560ce_characterizeImage_947526_29.stderr
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 730df4dbbcfdd214f72f824e09f2f89f with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bbe4e584-c2d3-4b33-a8d5-01f1535fec3e_calibrate_947526_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bbe4e584-c2d3-4b33-a8d5-01f1535fec3e_calibrate_947526_22.stderr
parsl.dataflow.dflow INFO: Task 1244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0492478b4edf95a71c5105d9bb02afc with result from task 1244
parsl.dataflow.dflow DEBUG: Task 1244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d8e8983d-9746-4125-82cf-b60bd5c947dc_calibrate_947528_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 1244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d8e8983d-9746-4125-82cf-b60bd5c947dc_calibrate_947528_43.stderr
parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fcd73849f285cd882359d8c31a0b523f with result from task 536
parsl.dataflow.dflow DEBUG: Task 536 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/12457ad7-3453-4387-a875-f0561db89018_calibrate_947526_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/12457ad7-3453-4387-a875-f0561db89018_calibrate_947526_32.stderr
parsl.dataflow.dflow INFO: Task 1162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c4ee8865756773dc045776039c1a217 with result from task 1162
parsl.dataflow.dflow DEBUG: Task 1162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1163 has memoization hash 7d7c014773bbe757e2175162f84d45a8
parsl.dataflow.memoization INFO: Task 1163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1163 try 0 launched on executor multi with executor id 1271
parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b979914a-9890-49f1-b889-8321e0dcc875_calibrate_947526_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b979914a-9890-49f1-b889-8321e0dcc875_calibrate_947526_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3902428e-069a-4ce4-8cda-20df43447e09_characterizeImage_947526_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/3902428e-069a-4ce4-8cda-20df43447e09_characterizeImage_947526_18.stderr
parsl.dataflow.dflow INFO: Task 1291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6adb2df28ffaf2fddb94a067c15da980 with result from task 1291
parsl.dataflow.dflow DEBUG: Task 1291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1292 has memoization hash c42aba5ed0e4ec1ee634e816a3ff9d3d
parsl.dataflow.memoization INFO: Task 1292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1292 try 0 launched on executor multi with executor id 1272
parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/4fb5ab97-1cd5-44a3-b555-be1143ff5051_calibrate_947524_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/4fb5ab97-1cd5-44a3-b555-be1143ff5051_calibrate_947524_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cbc8fa64-58f4-4a91-92cc-310ccc67ca6f_characterizeImage_947524_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/cbc8fa64-58f4-4a91-92cc-310ccc67ca6f_characterizeImage_947524_11.stderr
parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ab6b1f2196c78892790cb9a43bc538c with result from task 698
parsl.dataflow.dflow DEBUG: Task 698 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7fc7f8b9-6b16-46a2-a09c-f3b11ddffbd7_calibrate_947526_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/7fc7f8b9-6b16-46a2-a09c-f3b11ddffbd7_calibrate_947526_20.stderr
parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2a577cf072892beb94f9175a54045d9 with result from task 598
parsl.dataflow.dflow DEBUG: Task 598 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 599 has memoization hash af4c00cf192e54b80e1a9020bf020062
parsl.dataflow.memoization INFO: Task 599 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 1273
parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0116bfec-f837-4ad0-aa94-b9eb2b9a923e_calibrate_947422_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0116bfec-f837-4ad0-aa94-b9eb2b9a923e_calibrate_947422_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f2a3d5b6-1c64-4a78-82a3-6b6acbb72258_characterizeImage_947422_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/f2a3d5b6-1c64-4a78-82a3-6b6acbb72258_characterizeImage_947422_38.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f6b02605df3d3f56f84bd3cb5b4edcde with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d144c03a-6c83-4526-9504-6f6eff420907_calibrate_947524_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d144c03a-6c83-4526-9504-6f6eff420907_calibrate_947524_28.stderr
parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc0da4fdfd79631f575fe6e525ed7fdd with result from task 416
parsl.dataflow.dflow DEBUG: Task 416 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/1bb804de-a880-4d34-9dd4-67b4b0b3c93c_calibrate_947526_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/1bb804de-a880-4d34-9dd4-67b4b0b3c93c_calibrate_947526_44.stderr
parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0eae2efecf34ab4c00edd9951a231748 with result from task 251
parsl.dataflow.dflow DEBUG: Task 251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1c64395e-2d56-4a60-a986-442d376c2996_calibrate_947528_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1c64395e-2d56-4a60-a986-442d376c2996_calibrate_947528_16.stderr
parsl.dataflow.dflow INFO: Task 1153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50240ac343a399374b245e86ec4c22e1 with result from task 1153
parsl.dataflow.dflow DEBUG: Task 1153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1154 has memoization hash 4cd27e7398f39ac968b1c61cfb08cb07
parsl.dataflow.memoization INFO: Task 1154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1154 try 0 launched on executor multi with executor id 1274
parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e34cc8d7-61c9-4111-a5f5-1f184ec83272_calibrate_947528_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e34cc8d7-61c9-4111-a5f5-1f184ec83272_calibrate_947528_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/22db4a9d-cf42-4ea3-a0c6-33d177a9563e_characterizeImage_947528_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/22db4a9d-cf42-4ea3-a0c6-33d177a9563e_characterizeImage_947528_39.stderr
parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5371f8f97f1825c1c1e3e26383004092 with result from task 440
parsl.dataflow.dflow DEBUG: Task 440 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/8230e221-310f-4528-b4ae-da9774fae092_calibrate_947526_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/8230e221-310f-4528-b4ae-da9774fae092_calibrate_947526_3.stderr
parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e61721f296e81805897a2d1fb2f87ac7 with result from task 1103
parsl.dataflow.dflow DEBUG: Task 1103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/caf3eda1-2c8a-43f3-8937-90764862d38c_calibrate_947422_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/caf3eda1-2c8a-43f3-8937-90764862d38c_calibrate_947422_3.stderr
parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4befb1bc7469ca4ebd3b13ef27996a1e with result from task 917
parsl.dataflow.dflow DEBUG: Task 917 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/907305c3-39c2-4363-b82c-8af8c6766307_calibrate_947422_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/907305c3-39c2-4363-b82c-8af8c6766307_calibrate_947422_57.stderr
parsl.dataflow.dflow INFO: Task 1277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 648472687166fbcc241dd7ab5f60eda1 with result from task 1277
parsl.dataflow.dflow DEBUG: Task 1277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/6e10fd72-dd0f-4ebc-a49d-3d50f639ec7a_calibrate_947422_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/6e10fd72-dd0f-4ebc-a49d-3d50f639ec7a_calibrate_947422_30.stderr
2024-03-14 07:29:35 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:29:35 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:35 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:35 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7a51ccee83513eccc71d24742498199 with result from task 1235
parsl.dataflow.dflow DEBUG: Task 1235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e19ba07f-ade1-4ba6-b8f2-efa544947e23_calibrate_947523_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/e19ba07f-ade1-4ba6-b8f2-efa544947e23_calibrate_947523_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46851a11774bd0910274c0dd5bb944ca with result from task 887
parsl.dataflow.dflow DEBUG: Task 887 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9c037f8a-1b44-4cc8-ae95-1e5810a648b8_calibrate_947422_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/9c037f8a-1b44-4cc8-ae95-1e5810a648b8_calibrate_947422_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:36 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54d250247014943d330af91ff728503f with result from task 583
parsl.dataflow.dflow DEBUG: Task 583 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 4aab75484b4fe7748e80434adb5e4984
parsl.dataflow.memoization INFO: Task 584 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 1275
parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d9ff5257-3a2e-49fb-974f-77c80044b927_calibrate_947524_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d9ff5257-3a2e-49fb-974f-77c80044b927_calibrate_947524_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/61fdccf7-12fb-46f5-8116-2563a4e06eea_characterizeImage_947524_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/61fdccf7-12fb-46f5-8116-2563a4e06eea_characterizeImage_947524_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15d81b1ef038676313f472f3248ca225 with result from task 530
parsl.dataflow.dflow DEBUG: Task 530 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5576eb-5a89-4a22-b826-97f48c82538b_calibrate_947526_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6a5576eb-5a89-4a22-b826-97f48c82538b_calibrate_947526_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c56ce5f63e887c6757959b929ae9674 with result from task 1237
parsl.dataflow.dflow DEBUG: Task 1237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1238 has memoization hash 585db9f66ea0a02ccc6afa18135d4578
parsl.dataflow.memoization INFO: Task 1238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1238 try 0 launched on executor multi with executor id 1276
parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ec36b52c-f8fd-48d8-ae69-a6dd0e47672b_calibrate_947524_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ec36b52c-f8fd-48d8-ae69-a6dd0e47672b_calibrate_947524_32.stderr
parsl.dataflow.dflow INFO: Standard output for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7a9e012c-3549-4746-9848-0052e6c6614b_characterizeImage_947524_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/7a9e012c-3549-4746-9848-0052e6c6614b_characterizeImage_947524_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b61812b3d9533b82d95e9760942a37d with result from task 776
parsl.dataflow.dflow DEBUG: Task 776 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/916a2f2b-c6ec-44d5-9a09-fa2dbf80aa77_calibrate_947528_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/916a2f2b-c6ec-44d5-9a09-fa2dbf80aa77_calibrate_947528_58.stderr
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f832bf4ccf7b75300bec7a7dd62f3737 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 877ae5fa192578b13ed3d49e8336fddb
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e95c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 1277
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/08c35929-5707-4d24-866f-ad22582348e3_calibrate_947528_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/08c35929-5707-4d24-866f-ad22582348e3_calibrate_947528_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/49e60a25-8fe4-4f6c-9bec-c9c6455cfd75_characterizeImage_947528_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/49e60a25-8fe4-4f6c-9bec-c9c6455cfd75_characterizeImage_947528_25.stderr
parsl.dataflow.dflow INFO: Task 1301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4138f9a10a227b480f4ad71e655cf8e7 with result from task 1301
parsl.dataflow.dflow DEBUG: Task 1301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ad06130a-7d6d-45ce-8afe-172d87dda79c_calibrate_947418_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/ad06130a-7d6d-45ce-8afe-172d87dda79c_calibrate_947418_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:37 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21c778eb628220eef62309c7a5244cac with result from task 659
parsl.dataflow.dflow DEBUG: Task 659 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e3b459f2-ff9d-402f-8e1d-ec9f956cb1b8_calibrate_947526_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e3b459f2-ff9d-402f-8e1d-ec9f956cb1b8_calibrate_947526_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 254c5029e923516a951edea940eee6be with result from task 691
parsl.dataflow.dflow DEBUG: Task 691 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 22795818b60fa6fc9be26cb9b932465a
parsl.dataflow.memoization INFO: Task 692 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e37ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 1278
parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5be1f3e9-061e-4e10-b5fb-7d9e04001a4e_calibrate_947422_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5be1f3e9-061e-4e10-b5fb-7d9e04001a4e_calibrate_947422_62.stderr
parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb207f2b-3b32-4fdc-b642-36d142605abf_characterizeImage_947422_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/eb207f2b-3b32-4fdc-b642-36d142605abf_characterizeImage_947422_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7945bd772cde824b3e021f755740ac19 with result from task 1214
parsl.dataflow.dflow DEBUG: Task 1214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/319ef1be-964c-4d67-9e60-8b5d777b1838_calibrate_947524_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 1214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/319ef1be-964c-4d67-9e60-8b5d777b1838_calibrate_947524_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:38 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a533a378e2037bdf89b432cda5740cd with result from task 1082
parsl.dataflow.dflow DEBUG: Task 1082 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/210c0bcd-1ad5-4751-b450-10006654d114_calibrate_947524_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/210c0bcd-1ad5-4751-b450-10006654d114_calibrate_947524_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80aab61449ac0e29f12229325f5104ac with result from task 1142
parsl.dataflow.dflow DEBUG: Task 1142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/89d0cbe6-f8ad-4a56-922a-84eb6b491f7f_calibrate_947528_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 1142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/89d0cbe6-f8ad-4a56-922a-84eb6b491f7f_calibrate_947528_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:39 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc61103bcc9867cbc6a28f07899d705f with result from task 1264
parsl.dataflow.dflow DEBUG: Task 1264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1265 has memoization hash 2127346dae49a99c296d6dc53b72d3b3
parsl.dataflow.memoization INFO: Task 1265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1265 try 0 launched on executor multi with executor id 1279
parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/eb925286-981e-4775-a89a-972cba0c2437_calibrate_947422_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/eb925286-981e-4775-a89a-972cba0c2437_calibrate_947422_52.stderr
parsl.dataflow.dflow INFO: Standard output for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/529bcb75-2330-471c-ba2b-3e632b97e7be_characterizeImage_947422_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/529bcb75-2330-471c-ba2b-3e632b97e7be_characterizeImage_947422_52.stderr
parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 662ec108f0cf94a8da3e6cd7baebcf27 with result from task 308
parsl.dataflow.dflow DEBUG: Task 308 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/55e4b08b-ea04-4772-8f32-5e51d2dde373_calibrate_947523_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/55e4b08b-ea04-4772-8f32-5e51d2dde373_calibrate_947523_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 501021f75606a54bd6891ba0c0dbd784 with result from task 1127
parsl.dataflow.dflow DEBUG: Task 1127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e14f585c-2c87-45f1-975e-afbe7965e0fb_calibrate_947422_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/e14f585c-2c87-45f1-975e-afbe7965e0fb_calibrate_947422_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:40 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a87878dedd3150deeb795fe12d19cd0 with result from task 941
parsl.dataflow.dflow DEBUG: Task 941 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/6165cf3f-2cd5-4a1e-b15a-aac4d716347f_calibrate_947524_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/6165cf3f-2cd5-4a1e-b15a-aac4d716347f_calibrate_947524_19.stderr
parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5eecf8333057051e0131e5b4c454693c with result from task 302
parsl.dataflow.dflow DEBUG: Task 302 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b0a5c6e5-b220-4364-841e-0f11ef28230f_calibrate_947526_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b0a5c6e5-b220-4364-841e-0f11ef28230f_calibrate_947526_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7bea4dd1d3a8db23d9e4ec77199ceb9a with result from task 962
parsl.dataflow.dflow DEBUG: Task 962 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/97bfa387-69c5-4a87-a4a6-286a46ca081d_calibrate_947528_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/97bfa387-69c5-4a87-a4a6-286a46ca081d_calibrate_947528_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 576c54bb72343ccd377c64f426554228 with result from task 1178
parsl.dataflow.dflow DEBUG: Task 1178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/020ad163-32fc-4b2b-af8b-1959a08f765b_calibrate_947526_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/020ad163-32fc-4b2b-af8b-1959a08f765b_calibrate_947526_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f0b4bdbbc969b9efcba08ed738b3416 with result from task 1037
parsl.dataflow.dflow DEBUG: Task 1037 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/5b1fe438-0815-4455-b973-73a1b9ecd706_calibrate_947524_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/5b1fe438-0815-4455-b973-73a1b9ecd706_calibrate_947524_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58f3caba6f5ab022bac70ce7dd081f0d with result from task 995
parsl.dataflow.dflow DEBUG: Task 995 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/338286b2-94b3-4005-9f04-86a7dd2c2b91_calibrate_947420_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/338286b2-94b3-4005-9f04-86a7dd2c2b91_calibrate_947420_17.stderr
parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd6f1d5b2c1c3e6b2870b3892028852e with result from task 653
parsl.dataflow.dflow DEBUG: Task 653 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/09669497-a231-46b0-9c97-f238ad76aacd_calibrate_947526_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/09669497-a231-46b0-9c97-f238ad76aacd_calibrate_947526_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aeaf0ea8c0f08296ef1942c05d0596aa with result from task 220
parsl.dataflow.dflow DEBUG: Task 220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 221 has memoization hash f2bbfe8b16ab63473c72aeb73a78a2b5
parsl.dataflow.memoization INFO: Task 221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 1280
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/4a8b03cc-9a17-442c-873e-256cd3ae2f07_calibrate_947526_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/4a8b03cc-9a17-442c-873e-256cd3ae2f07_calibrate_947526_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c5e740b8-ec14-4bcd-b02a-2b4adf5dfbf1_characterizeImage_947526_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/c5e740b8-ec14-4bcd-b02a-2b4adf5dfbf1_characterizeImage_947526_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:41 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e4fb67619d77f85c20af1528fa393ca with result from task 280
parsl.dataflow.dflow DEBUG: Task 280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 931dbe193db00d07ed8031fea9bd5a53
parsl.dataflow.memoization INFO: Task 281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f00c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 1281
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/b88655aa-8827-46b4-a9a4-b7c95206afe2_calibrate_947420_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/b88655aa-8827-46b4-a9a4-b7c95206afe2_calibrate_947420_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0c958d5b-9f5e-44de-a1cd-06850071ebcc_characterizeImage_947420_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/0c958d5b-9f5e-44de-a1cd-06850071ebcc_characterizeImage_947420_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc3aa85ea09150e5f684e19a286016e with result from task 764
parsl.dataflow.dflow DEBUG: Task 764 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0c3fa93a-10a6-4150-be9b-ae23d4e63a85_calibrate_947528_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0c3fa93a-10a6-4150-be9b-ae23d4e63a85_calibrate_947528_60.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7539cde4920e785540958a01a3e74e3f with result from task 779
parsl.dataflow.dflow DEBUG: Task 779 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b3f12e7b-5056-4d39-a231-92f29a9a1787_calibrate_947523_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/b3f12e7b-5056-4d39-a231-92f29a9a1787_calibrate_947523_61.stderr
parsl.dataflow.dflow INFO: Task 1166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd3393deaad9c929e82e1a7c7cdc2bfb with result from task 1166
parsl.dataflow.dflow DEBUG: Task 1166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/36beb3db-f089-45a3-b1e4-118f5e9ba596_calibrate_947526_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 1166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/36beb3db-f089-45a3-b1e4-118f5e9ba596_calibrate_947526_23.stderr
parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe6dd44c3080e22e8b2262ee534d1933 with result from task 880
parsl.dataflow.dflow DEBUG: Task 880 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 881 has memoization hash ccdbcea6d2b7b595f3e2c30480c5691f
parsl.dataflow.memoization INFO: Task 881 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 1282
parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e447b4a9-93e7-445e-a190-12ab3487d9d5_calibrate_947524_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e447b4a9-93e7-445e-a190-12ab3487d9d5_calibrate_947524_18.stderr
parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b239acb1-3828-4a4a-83c1-59b662f7e91b_characterizeImage_947524_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/b239acb1-3828-4a4a-83c1-59b662f7e91b_characterizeImage_947524_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f1f7c3178ac63df94710c875756d01a with result from task 662
parsl.dataflow.dflow DEBUG: Task 662 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6f793069-dda6-4a10-94c9-8c8b50b464ea_calibrate_947526_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/6f793069-dda6-4a10-94c9-8c8b50b464ea_calibrate_947526_8.stderr
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e17f0a7819761b2a56fff1b51534186 with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/27561a78-0b5d-4f12-97d9-05f17c91ed2f_calibrate_947526_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/27561a78-0b5d-4f12-97d9-05f17c91ed2f_calibrate_947526_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:42 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 461da71da082bcf18d7ebb2ad3f850d9 with result from task 754
parsl.dataflow.dflow DEBUG: Task 754 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 6947697c725ac892b4dd9866a848faa0
parsl.dataflow.memoization INFO: Task 755 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e77d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 1283
parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a3b662a-a1d9-4da7-af24-8ba5fa37e54b_calibrate_947523_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a3b662a-a1d9-4da7-af24-8ba5fa37e54b_calibrate_947523_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8659ec58-f07a-4513-85b3-d3bf00214833_characterizeImage_947523_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947523/8659ec58-f07a-4513-85b3-d3bf00214833_characterizeImage_947523_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 337e0ee42044b841f6ce72d658da9544 with result from task 818
parsl.dataflow.dflow DEBUG: Task 818 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/58ef388a-48fc-44a8-91be-ef08a2bbf375_calibrate_947524_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/58ef388a-48fc-44a8-91be-ef08a2bbf375_calibrate_947524_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19adb1388f86c9b3f77f479c95fab4e3 with result from task 563
parsl.dataflow.dflow DEBUG: Task 563 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f2dea5e2-0a5f-47f3-8553-5f0cb2ceb844_calibrate_947528_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f2dea5e2-0a5f-47f3-8553-5f0cb2ceb844_calibrate_947528_36.stderr
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9b0028b46246dc0627565839955df762 with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/74d07640-cf14-4990-88de-2aabeeb6c4e3_calibrate_947524_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/74d07640-cf14-4990-88de-2aabeeb6c4e3_calibrate_947524_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:43 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81959b16ce6d66eb39200b40b4efad6d with result from task 533
parsl.dataflow.dflow DEBUG: Task 533 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d784cef2-f283-4b9b-b813-adf64a359d9f_calibrate_947528_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d784cef2-f283-4b9b-b813-adf64a359d9f_calibrate_947528_52.stderr
parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c4b2e796a9a84db0f7faf2569583513 with result from task 650
parsl.dataflow.dflow DEBUG: Task 650 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/76aae12f-d989-4ea0-9dd6-77126031994d_calibrate_947422_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/76aae12f-d989-4ea0-9dd6-77126031994d_calibrate_947422_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:44 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f4baa6aa4b731c9c7c2198c8353ce6c with result from task 788
parsl.dataflow.dflow DEBUG: Task 788 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/3e177025-4ecd-490a-88d5-0efac89cabdf_calibrate_947524_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/3e177025-4ecd-490a-88d5-0efac89cabdf_calibrate_947524_16.stderr
parsl.dataflow.dflow INFO: Task 1151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 063161a879270c6b4e6623f6728e977c with result from task 1151
parsl.dataflow.dflow DEBUG: Task 1151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/33034da0-8562-4b6b-b1b4-2afd6e59168d_calibrate_947526_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/33034da0-8562-4b6b-b1b4-2afd6e59168d_calibrate_947526_48.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47cb65d3d287cd5476b353d4f25db99f with result from task 1115
parsl.dataflow.dflow DEBUG: Task 1115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ca27478-5124-42df-8e63-6718b8e10b3f_calibrate_947526_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/3ca27478-5124-42df-8e63-6718b8e10b3f_calibrate_947526_7.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:45 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51882b9a1125bf994603027584414b40 with result from task 1043
parsl.dataflow.dflow DEBUG: Task 1043 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d8a86879-7497-448a-9d78-a7354f5bc447_calibrate_947526_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/d8a86879-7497-448a-9d78-a7354f5bc447_calibrate_947526_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7724eb64d16d4d8d43cf87e6c2f1e6e8 with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a4eefb8b-c9c9-44d4-9327-7fb95bf42f04_calibrate_947528_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a4eefb8b-c9c9-44d4-9327-7fb95bf42f04_calibrate_947528_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:46 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:47 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:47 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:47 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:47 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:47 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:29:47 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5970084c48affbeef951d319a1198aea with result from task 1184
parsl.dataflow.dflow DEBUG: Task 1184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/23c0d55d-937c-4e40-9ab4-c133e76b7c43_calibrate_947528_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 1184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/23c0d55d-937c-4e40-9ab4-c133e76b7c43_calibrate_947528_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033372 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 76
parsl.dataflow.strategy DEBUG: Executor multi has 76 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 76
parsl.dataflow.strategy DEBUG: Executor multi has 76 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 1283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0f04391dd4ebc96aba0cc2ed8d1fcf5 with result from task 1283
parsl.dataflow.dflow DEBUG: Task 1283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/39217143-35e1-4a8c-8a95-c8d3f1251b74_calibrate_947523_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 1283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/39217143-35e1-4a8c-8a95-c8d3f1251b74_calibrate_947523_25.stderr
parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf51c3583b48bcd7579290cf1c85719c with result from task 958
parsl.dataflow.dflow DEBUG: Task 958 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 82de86439935e24db3d23b946de32095
parsl.dataflow.memoization INFO: Task 959 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 1284
parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/85f1ddd5-07c0-4a7a-b281-ff68917d2781_calibrate_947524_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/85f1ddd5-07c0-4a7a-b281-ff68917d2781_calibrate_947524_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a7b24683-7b68-4901-9e4f-9f718e6dd247_characterizeImage_947524_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/a7b24683-7b68-4901-9e4f-9f718e6dd247_characterizeImage_947524_38.stderr
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22c72a25421d478a177877b27e05b52d with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d41f3698-1893-450d-bed3-b4c0d98a6e39_calibrate_947524_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d41f3698-1893-450d-bed3-b4c0d98a6e39_calibrate_947524_37.stderr
parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4451fec005ef3803f2b6c27e1cf95fbb with result from task 209
parsl.dataflow.dflow DEBUG: Task 209 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae6cb67-17d4-4a16-a280-c85ec376dbd0_calibrate_947524_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/aae6cb67-17d4-4a16-a280-c85ec376dbd0_calibrate_947524_17.stderr
parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 252fa90337c2b6d8c17ee7e218efc68b with result from task 935
parsl.dataflow.dflow DEBUG: Task 935 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a759ce46-4b9e-493e-8747-639784d38beb_calibrate_947526_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/a759ce46-4b9e-493e-8747-639784d38beb_calibrate_947526_35.stderr
parsl.dataflow.dflow INFO: Task 1187 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b71138f056c14c41810744481cd6469d with result from task 1187
parsl.dataflow.dflow DEBUG: Task 1187 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fb04abc3-1bac-44a8-9eb6-b25685d3a988_calibrate_947524_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 1187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/fb04abc3-1bac-44a8-9eb6-b25685d3a988_calibrate_947524_35.stderr
parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2aeb1acd881931c90b7c35ab88a677e2 with result from task 746
parsl.dataflow.dflow DEBUG: Task 746 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a134df97-00d6-4165-bff3-76012a50f696_calibrate_947524_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/a134df97-00d6-4165-bff3-76012a50f696_calibrate_947524_45.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 70
parsl.dataflow.strategy DEBUG: Executor multi has 70 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 052d9585a9d325085d113234f7f37a37 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0ac4511d-2cc6-4ecd-b946-a60f1ea814df_calibrate_947526_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/0ac4511d-2cc6-4ecd-b946-a60f1ea814df_calibrate_947526_24.stderr
parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc05375195139bd140c244a7e832a2a with result from task 632
parsl.dataflow.dflow DEBUG: Task 632 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5ba3e220-22b1-4911-ac23-0fa983f40fbb_calibrate_947523_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/5ba3e220-22b1-4911-ac23-0fa983f40fbb_calibrate_947523_58.stderr
parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93aab17e34b12055f9689edb5357a073 with result from task 260
parsl.dataflow.dflow DEBUG: Task 260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a408b5cf-0940-4a5f-9d04-130fd3e53855_calibrate_947528_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a408b5cf-0940-4a5f-9d04-130fd3e53855_calibrate_947528_45.stderr
parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8176101254207727b3358d6f32302e5b with result from task 512
parsl.dataflow.dflow DEBUG: Task 512 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8fc47b4d-8f34-4e26-865b-e4a13093de1f_calibrate_947528_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8fc47b4d-8f34-4e26-865b-e4a13093de1f_calibrate_947528_7.stderr
parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb4cfedc0d0755865005865c6f2ebb49 with result from task 344
parsl.dataflow.dflow DEBUG: Task 344 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e8d9209b-da58-4040-a427-d3556b64b616_calibrate_947418_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/e8d9209b-da58-4040-a427-d3556b64b616_calibrate_947418_25.stderr
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 152a1ebb6acc7a9d0e7be0b2b995dfb8 with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ba50c37f-b222-49c0-a915-e300741a015c_calibrate_947524_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ba50c37f-b222-49c0-a915-e300741a015c_calibrate_947524_60.stderr
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33aec193b3bbb80f5cbae712273e70ae with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bffd4049-1b00-4983-8ad0-6422c0ca0d29_calibrate_947526_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/bffd4049-1b00-4983-8ad0-6422c0ca0d29_calibrate_947526_40.stderr
parsl.dataflow.dflow INFO: Task 1160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8028f1a903ec7518b8f19863a60e1c90 with result from task 1160
parsl.dataflow.dflow DEBUG: Task 1160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/3cbd35f9-af8c-4d12-ac4a-77a97f6b4d72_calibrate_947528_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/3cbd35f9-af8c-4d12-ac4a-77a97f6b4d72_calibrate_947528_29.stderr
parsl.dataflow.dflow INFO: Task 1252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 728f73f08291f259dc9d8684d31ee60f with result from task 1252
parsl.dataflow.dflow DEBUG: Task 1252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1253 has memoization hash 94ca102ba7ce7b13a59400908ad89f10
parsl.dataflow.memoization INFO: Task 1253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8b0c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1253 try 0 launched on executor multi with executor id 1285
parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b1e03d46-9207-4bcd-9018-258b5d115403_calibrate_947524_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b1e03d46-9207-4bcd-9018-258b5d115403_calibrate_947524_50.stderr
parsl.dataflow.dflow INFO: Standard output for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/120a6262-57ee-437a-9b97-e5e12bb1bc58_characterizeImage_947524_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/120a6262-57ee-437a-9b97-e5e12bb1bc58_characterizeImage_947524_50.stderr
parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9f72c10dbba5827c074bc3699552230 with result from task 317
parsl.dataflow.dflow DEBUG: Task 317 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/017e779a-10fb-4e2b-8240-74ebbc7c4a7c_calibrate_947526_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/017e779a-10fb-4e2b-8240-74ebbc7c4a7c_calibrate_947526_6.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 61
parsl.dataflow.strategy DEBUG: Executor multi has 61 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53a8a7812d2c6d0a01aa4f1de71c0722 with result from task 410
parsl.dataflow.dflow DEBUG: Task 410 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e5e41e8a-0ac6-4e1b-8d17-75edd607de78_calibrate_947526_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/e5e41e8a-0ac6-4e1b-8d17-75edd607de78_calibrate_947526_28.stderr
parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8bf6c75fb1ab803e93f1750464ae1bad with result from task 461
parsl.dataflow.dflow DEBUG: Task 461 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/c5290151-0427-4946-9278-5dc73c454987_calibrate_947526_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/c5290151-0427-4946-9278-5dc73c454987_calibrate_947526_25.stderr
parsl.dataflow.dflow INFO: Task 1226 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44525d9425fb854f4cc672e0a2faf434 with result from task 1226
parsl.dataflow.dflow DEBUG: Task 1226 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f2c07152-6a29-4a15-8f51-43a10209ff3e_calibrate_947524_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 1226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f2c07152-6a29-4a15-8f51-43a10209ff3e_calibrate_947524_21.stderr
parsl.dataflow.dflow INFO: Task 1120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1aa2e21d877d5fe72f7f98eacc2c0e with result from task 1120
parsl.dataflow.dflow DEBUG: Task 1120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1121 has memoization hash 07b5ba7872161bb0f53254c1c465f7da
parsl.dataflow.memoization INFO: Task 1121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8c37c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1121 try 0 launched on executor multi with executor id 1286
parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/ab7cd52d-a07d-46f5-b511-c938fe689ce2_calibrate_947526_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/ab7cd52d-a07d-46f5-b511-c938fe689ce2_calibrate_947526_45.stderr
parsl.dataflow.dflow INFO: Standard output for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/eaa03c38-3a1a-4f5e-ba79-3424d06ddece_characterizeImage_947526_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/eaa03c38-3a1a-4f5e-ba79-3424d06ddece_characterizeImage_947526_45.stderr
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f969fda4f595861671a8d6d6cecb399c with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash a21e316185f702a3c9bd4bbff0108e65
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8e94360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 1287
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d5808a35-8964-447b-8098-7b449935e370_calibrate_947528_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d5808a35-8964-447b-8098-7b449935e370_calibrate_947528_19.stderr
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/337c0eff-5310-4e6a-aead-449b628c4b1e_characterizeImage_947528_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/337c0eff-5310-4e6a-aead-449b628c4b1e_characterizeImage_947528_19.stderr
parsl.dataflow.dflow INFO: Task 1295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f667699996d7a9d4dffcce83c8b46210 with result from task 1295
parsl.dataflow.dflow DEBUG: Task 1295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f92218b2-8bd6-4d95-99c3-a11edcf5fcb9_calibrate_947528_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/f92218b2-8bd6-4d95-99c3-a11edcf5fcb9_calibrate_947528_41.stderr
parsl.dataflow.dflow INFO: Task 1268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a226239893fea9a5a845544bc2057383 with result from task 1268
parsl.dataflow.dflow DEBUG: Task 1268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1d1966ac-253e-4646-8417-1958ca6a130c_calibrate_947528_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 1268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/1d1966ac-253e-4646-8417-1958ca6a130c_calibrate_947528_48.stderr
parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73cee65736590aa768ee97aa13cdba5f with result from task 1079
parsl.dataflow.dflow DEBUG: Task 1079 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0da68745-6865-4ebe-82bc-2098850d16e8_calibrate_947528_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/0da68745-6865-4ebe-82bc-2098850d16e8_calibrate_947528_12.stderr
parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a52dfecd7f158f234b1dacb4a3276253 with result from task 1004
parsl.dataflow.dflow DEBUG: Task 1004 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d8d11901-37e1-4436-8832-a980d6f79b78_calibrate_947524_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d8d11901-37e1-4436-8832-a980d6f79b78_calibrate_947524_3.stderr
parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d63d85589370d9bd0050b0c9871eff61 with result from task 710
parsl.dataflow.dflow DEBUG: Task 710 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/958d7d95-7b15-4126-8ff7-6403b7e115d6_calibrate_947526_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/958d7d95-7b15-4126-8ff7-6403b7e115d6_calibrate_947526_5.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 53
parsl.dataflow.strategy DEBUG: Executor multi has 53 active tasks, 109/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a2c8113ccf8b90c7ee749f3674b251c with result from task 446
parsl.dataflow.dflow DEBUG: Task 446 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/2e8a6bc9-56f4-4ff8-9113-850715aeb0ff_calibrate_947422_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/2e8a6bc9-56f4-4ff8-9113-850715aeb0ff_calibrate_947422_46.stderr
parsl.dataflow.dflow INFO: Task 1256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51afcbb1d14f112aad5367e598b3d40b with result from task 1256
parsl.dataflow.dflow DEBUG: Task 1256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e28f0e71-969f-4e16-b083-eb9593f4ce98_calibrate_947524_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e28f0e71-969f-4e16-b083-eb9593f4ce98_calibrate_947524_39.stderr
parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec76a77c9fc58db6960943d20723df7f with result from task 968
parsl.dataflow.dflow DEBUG: Task 968 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/30ac6cb7-f48b-4c73-9477-0a889d37deed_calibrate_947524_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/30ac6cb7-f48b-4c73-9477-0a889d37deed_calibrate_947524_40.stderr
parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba02893359489af24a4e6e6c060a274b with result from task 923
parsl.dataflow.dflow DEBUG: Task 923 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/06af0cd0-b7fb-4187-9426-2f82a7ab3c3a_calibrate_947526_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/06af0cd0-b7fb-4187-9426-2f82a7ab3c3a_calibrate_947526_51.stderr
parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ae48b8dc36a58cc23b8c2fc899d7d83 with result from task 613
parsl.dataflow.dflow DEBUG: Task 613 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 684fdb779cbf04a915a03f72e015ae55
parsl.dataflow.memoization INFO: Task 614 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8dcea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 1288
parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/72c14628-6091-442b-a066-3c5c6fb32876_calibrate_947422_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/72c14628-6091-442b-a066-3c5c6fb32876_calibrate_947422_39.stderr
parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/33306b88-d18f-4587-b457-b7f2a9185bbc_characterizeImage_947422_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/33306b88-d18f-4587-b457-b7f2a9185bbc_characterizeImage_947422_39.stderr
parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27545f2fc607e58111b2678ed7fea055 with result from task 1040
parsl.dataflow.dflow DEBUG: Task 1040 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/363943fd-9d46-4af0-b2cc-887d2f62351f_calibrate_947526_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/363943fd-9d46-4af0-b2cc-887d2f62351f_calibrate_947526_41.stderr
parsl.dataflow.dflow INFO: Task 1144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3793bde8e7b7cfdff03b60262ceba4be with result from task 1144
parsl.dataflow.dflow DEBUG: Task 1144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1145 has memoization hash 77ca6fc496e346fd6486dafe552cc48a
parsl.dataflow.memoization INFO: Task 1145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8a7d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1145 try 0 launched on executor multi with executor id 1289
parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a89c029b-bb38-48f2-872d-534d2e6c07e2_calibrate_947528_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a89c029b-bb38-48f2-872d-534d2e6c07e2_calibrate_947528_17.stderr
parsl.dataflow.dflow INFO: Standard output for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/88d63cb6-78cd-4229-b4b4-a312b235bef6_characterizeImage_947528_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/88d63cb6-78cd-4229-b4b4-a312b235bef6_characterizeImage_947528_17.stderr
2024-03-14 07:30:07 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:30:07 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:07 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:08 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:08 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:08 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:08 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:08 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:08 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:09 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e806ed525afbc62bb5ce9a1de3330e6 with result from task 1012
parsl.dataflow.dflow DEBUG: Task 1012 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 27ac3833df71d89ff643c46cbe4f835a
parsl.dataflow.memoization INFO: Task 1013 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1290
parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f8f0f3c-c89f-4ee9-bdb9-248955d36b0f_calibrate_947528_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f8f0f3c-c89f-4ee9-bdb9-248955d36b0f_calibrate_947528_38.stderr
parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/84655c21-6494-43a7-9647-95db66ee33ac_characterizeImage_947528_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/84655c21-6494-43a7-9647-95db66ee33ac_characterizeImage_947528_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1202 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fab8834d14ec8121129b43454d1cc47b with result from task 1202
parsl.dataflow.dflow DEBUG: Task 1202 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5705427f-2dba-476c-aacf-f4f79289fad2_calibrate_947528_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 1202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/5705427f-2dba-476c-aacf-f4f79289fad2_calibrate_947528_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53a9c98cd0b186c1bcec4461302dff5e with result from task 199
parsl.dataflow.dflow DEBUG: Task 199 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 26e5d94a91ca24658c335548390f6a96
parsl.dataflow.memoization INFO: Task 200 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 1291
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79398958-a42e-436f-83e7-6f55265294e3_calibrate_947418_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79398958-a42e-436f-83e7-6f55265294e3_calibrate_947418_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1f1786ee-a348-46c7-8db0-4ffe50ab5818_characterizeImage_947418_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/1f1786ee-a348-46c7-8db0-4ffe50ab5818_characterizeImage_947418_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:10 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc130e6b25cb0b177d4714b45b00ba8c with result from task 1232
parsl.dataflow.dflow DEBUG: Task 1232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/12dc46e1-08d7-4f2f-9514-f245557f77ab_calibrate_947528_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/12dc46e1-08d7-4f2f-9514-f245557f77ab_calibrate_947528_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 1220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4766f90f1be7cf66f674896d559880 with result from task 1220
parsl.dataflow.dflow DEBUG: Task 1220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/93e53ec7-34df-45bd-8332-a100a15e53cf_calibrate_947526_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/93e53ec7-34df-45bd-8332-a100a15e53cf_calibrate_947526_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e037f9ed7dc5644c3ab5cfe07d51b2ec with result from task 257
parsl.dataflow.dflow DEBUG: Task 257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/92cadb84-642a-474e-a712-230a0a0b83da_calibrate_947528_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/92cadb84-642a-474e-a712-230a0a0b83da_calibrate_947528_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:11 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 931dbe193db00d07ed8031fea9bd5a53 with result from task 281
parsl.dataflow.dflow DEBUG: Task 281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/b88655aa-8827-46b4-a9a4-b7c95206afe2_calibrate_947420_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/b88655aa-8827-46b4-a9a4-b7c95206afe2_calibrate_947420_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 255ccbc055311f76526de4acebb53c74 with result from task 253
parsl.dataflow.dflow DEBUG: Task 253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 254 has memoization hash c3e396af91964286a192316d1dc16287
parsl.dataflow.memoization INFO: Task 254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 1292
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/494efb28-18f9-4a1b-b438-0d0955748c08_calibrate_947418_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/494efb28-18f9-4a1b-b438-0d0955748c08_calibrate_947418_16.stderr
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/22e5da8f-b83d-43da-ba9d-d17c1547c7bf_characterizeImage_947418_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/22e5da8f-b83d-43da-ba9d-d17c1547c7bf_characterizeImage_947418_16.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:12 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:13 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ca0d8f1372ffc4b52e1b62c82130550 with result from task 1228
parsl.dataflow.dflow DEBUG: Task 1228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1229 has memoization hash 9b1295f628d917c999eb5923f813152d
parsl.dataflow.memoization INFO: Task 1229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1229 try 0 launched on executor multi with executor id 1293
parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c2b6890a-9a2f-4618-8803-885abaa4ea43_calibrate_947528_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c2b6890a-9a2f-4618-8803-885abaa4ea43_calibrate_947528_20.stderr
parsl.dataflow.dflow INFO: Standard output for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1d2f75f5-111f-45a3-809c-e898099845c3_characterizeImage_947528_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/1d2f75f5-111f-45a3-809c-e898099845c3_characterizeImage_947528_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7355a57899a4a5db1511d44b9586bd32 with result from task 991
parsl.dataflow.dflow DEBUG: Task 991 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 461327b2213ef77c2e22f67a06a55f22
parsl.dataflow.memoization INFO: Task 992 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 1294
parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b10a45fd-b7fa-4f8b-b977-c30b44f3c936_calibrate_947524_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b10a45fd-b7fa-4f8b-b977-c30b44f3c936_calibrate_947524_25.stderr
parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/5d5260e0-981f-4ddd-933a-f54628745362_characterizeImage_947524_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/5d5260e0-981f-4ddd-933a-f54628745362_characterizeImage_947524_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033372 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033372 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033372,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:14 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b9d90aeb305d995556e5bd135048db6 with result from task 1148
parsl.dataflow.dflow DEBUG: Task 1148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7e43ca49-e3ff-4ff7-aede-ce6810c57a25_calibrate_947524_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 1148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7e43ca49-e3ff-4ff7-aede-ce6810c57a25_calibrate_947524_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bfec3034b83a9c62f2eaa2bfb202b1ee with result from task 1027
parsl.dataflow.dflow DEBUG: Task 1027 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 714740b9097f2239217617b89b812644
parsl.dataflow.memoization INFO: Task 1028 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 1295
parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dd173ead-dcd3-4eca-8df4-b654e84fba0a_calibrate_947418_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dd173ead-dcd3-4eca-8df4-b654e84fba0a_calibrate_947418_11.stderr
parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5e3343d9-53eb-40b3-89ff-310a75d49d24_characterizeImage_947418_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/5e3343d9-53eb-40b3-89ff-310a75d49d24_characterizeImage_947418_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:15 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b911d1d95eebd6a2013ad7cb3221d43 with result from task 1262
parsl.dataflow.dflow DEBUG: Task 1262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7d149435-895b-428e-a8cf-9685cd943f58_calibrate_947524_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/7d149435-895b-428e-a8cf-9685cd943f58_calibrate_947524_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a29bd8b6494b94ff0202b5d9dc0991e with result from task 293
parsl.dataflow.dflow DEBUG: Task 293 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f87bb675-adbd-453a-8d3d-c6a4f1f75825_calibrate_947524_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/f87bb675-adbd-453a-8d3d-c6a4f1f75825_calibrate_947524_8.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00d90054c9dac77c60b575a2ccdee1fb with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e69c6151-35fc-4286-99e0-c0234aa5f17f_calibrate_947528_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e69c6151-35fc-4286-99e0-c0234aa5f17f_calibrate_947528_61.stderr
parsl.dataflow.dflow INFO: Task 1238 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 585db9f66ea0a02ccc6afa18135d4578 with result from task 1238
parsl.dataflow.dflow DEBUG: Task 1238 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ec36b52c-f8fd-48d8-ae69-a6dd0e47672b_calibrate_947524_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 1238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/ec36b52c-f8fd-48d8-ae69-a6dd0e47672b_calibrate_947524_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:16 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22795818b60fa6fc9be26cb9b932465a with result from task 692
parsl.dataflow.dflow DEBUG: Task 692 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5be1f3e9-061e-4e10-b5fb-7d9e04001a4e_calibrate_947422_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/5be1f3e9-061e-4e10-b5fb-7d9e04001a4e_calibrate_947422_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:17 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 634ba6fcf1fd23848819e8ec6c8ee6ce with result from task 341
parsl.dataflow.dflow DEBUG: Task 341 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b9c264ad-8995-44e2-93be-d78465a5f778_calibrate_947524_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b9c264ad-8995-44e2-93be-d78465a5f778_calibrate_947524_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:18 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:19 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:20 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:20 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d6659e0442146e07c1ef8010fd5ec8db with result from task 1130
parsl.dataflow.dflow DEBUG: Task 1130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f2354ac5-6371-4ab7-b540-adc0dddf65af_calibrate_947526_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 1130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/f2354ac5-6371-4ab7-b540-adc0dddf65af_calibrate_947526_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:20 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:20 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 35
parsl.dataflow.strategy DEBUG: Executor multi has 35 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 35
parsl.dataflow.strategy DEBUG: Executor multi has 35 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 1265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2127346dae49a99c296d6dc53b72d3b3 with result from task 1265
parsl.dataflow.dflow DEBUG: Task 1265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/eb925286-981e-4775-a89a-972cba0c2437_calibrate_947422_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 1265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/eb925286-981e-4775-a89a-972cba0c2437_calibrate_947422_52.stderr
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 877ae5fa192578b13ed3d49e8336fddb with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/08c35929-5707-4d24-866f-ad22582348e3_calibrate_947528_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/08c35929-5707-4d24-866f-ad22582348e3_calibrate_947528_25.stderr
parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e669b8fb2d2bb4f2474a6e25d190517 with result from task 236
parsl.dataflow.dflow DEBUG: Task 236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/182b599e-1182-47c7-b22d-585f80cc1e5d_calibrate_947526_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/182b599e-1182-47c7-b22d-585f80cc1e5d_calibrate_947526_4.stderr
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 842 try 0 failed
parsl.dataflow.dflow INFO: Task 842 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 3ce3e6645a807a73da36bb2cb694d99c
parsl.dataflow.memoization INFO: Task 842 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d098a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 842 try 1 launched on executor multi with executor id 1296
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stderr
parsl.dataflow.dflow INFO: Task 1154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd27e7398f39ac968b1c61cfb08cb07 with result from task 1154
parsl.dataflow.dflow DEBUG: Task 1154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e34cc8d7-61c9-4111-a5f5-1f184ec83272_calibrate_947528_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 1154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/e34cc8d7-61c9-4111-a5f5-1f184ec83272_calibrate_947528_39.stderr
parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8449fcb09194f5cda944a1385fa93afc with result from task 926
parsl.dataflow.dflow DEBUG: Task 926 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/746cd0a9-25b2-4480-bdd7-ec073e1157d2_calibrate_947524_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/746cd0a9-25b2-4480-bdd7-ec073e1157d2_calibrate_947524_6.stderr
parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0a676506ab7fc2178914fd5a1243966 with result from task 193
parsl.dataflow.dflow DEBUG: Task 193 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 4f41ed2b07e4b5a71ad978135435236a
parsl.dataflow.memoization INFO: Task 194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ec3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 1297
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/75c3f1e8-b00b-42dc-b545-7108fdcf00f0_calibrate_947526_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/75c3f1e8-b00b-42dc-b545-7108fdcf00f0_calibrate_947526_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fcdfec6d-ed95-4b4f-8c62-0577fba8f901_characterizeImage_947526_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947526/fcdfec6d-ed95-4b4f-8c62-0577fba8f901_characterizeImage_947526_30.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 30
parsl.dataflow.strategy DEBUG: Executor multi has 30 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6947697c725ac892b4dd9866a848faa0 with result from task 755
parsl.dataflow.dflow DEBUG: Task 755 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a3b662a-a1d9-4da7-af24-8ba5fa37e54b_calibrate_947523_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947523/9a3b662a-a1d9-4da7-af24-8ba5fa37e54b_calibrate_947523_17.stderr
parsl.dataflow.dflow INFO: Task 1292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c42aba5ed0e4ec1ee634e816a3ff9d3d with result from task 1292
parsl.dataflow.dflow DEBUG: Task 1292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/4fb5ab97-1cd5-44a3-b555-be1143ff5051_calibrate_947524_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/4fb5ab97-1cd5-44a3-b555-be1143ff5051_calibrate_947524_11.stderr
parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb4b7fbacbe3acf0ecb1d8ec5996a29 with result from task 976
parsl.dataflow.dflow DEBUG: Task 976 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 542adbc2c4ed1574c08dc268dbae3ef1
parsl.dataflow.memoization INFO: Task 977 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ba25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 1298
parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fd129370-5be8-44e4-9e51-9e34b8ea1993_calibrate_947418_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fd129370-5be8-44e4-9e51-9e34b8ea1993_calibrate_947418_24.stderr
parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d50b4a27-056b-48e9-9725-575aa49e2b40_characterizeImage_947418_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/d50b4a27-056b-48e9-9725-575aa49e2b40_characterizeImage_947418_24.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 28
parsl.dataflow.strategy DEBUG: Executor multi has 28 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccdbcea6d2b7b595f3e2c30480c5691f with result from task 881
parsl.dataflow.dflow DEBUG: Task 881 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e447b4a9-93e7-445e-a190-12ab3487d9d5_calibrate_947524_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/e447b4a9-93e7-445e-a190-12ab3487d9d5_calibrate_947524_18.stderr
parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af4c00cf192e54b80e1a9020bf020062 with result from task 599
parsl.dataflow.dflow DEBUG: Task 599 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0116bfec-f837-4ad0-aa94-b9eb2b9a923e_calibrate_947422_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/0116bfec-f837-4ad0-aa94-b9eb2b9a923e_calibrate_947422_38.stderr
parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4aab75484b4fe7748e80434adb5e4984 with result from task 584
parsl.dataflow.dflow DEBUG: Task 584 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d9ff5257-3a2e-49fb-974f-77c80044b927_calibrate_947524_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/d9ff5257-3a2e-49fb-974f-77c80044b927_calibrate_947524_30.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 25
parsl.dataflow.strategy DEBUG: Executor multi has 25 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2bbfe8b16ab63473c72aeb73a78a2b5 with result from task 221
parsl.dataflow.dflow DEBUG: Task 221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/4a8b03cc-9a17-442c-873e-256cd3ae2f07_calibrate_947526_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/4a8b03cc-9a17-442c-873e-256cd3ae2f07_calibrate_947526_38.stderr
2024-03-14 07:30:40 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:30:40 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7d7c014773bbe757e2175162f84d45a8 with result from task 1163
parsl.dataflow.dflow DEBUG: Task 1163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b979914a-9890-49f1-b889-8321e0dcc875_calibrate_947526_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 1163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/b979914a-9890-49f1-b889-8321e0dcc875_calibrate_947526_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:40 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:40 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:40 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:41 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2c3e9ce99782b0a9345ed582cebed4d with result from task 1274
parsl.dataflow.dflow DEBUG: Task 1274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/86c99b0a-8a37-4b13-9d77-904e0418bdb7_calibrate_947528_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 1274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/86c99b0a-8a37-4b13-9d77-904e0418bdb7_calibrate_947528_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 684fdb779cbf04a915a03f72e015ae55 with result from task 614
parsl.dataflow.dflow DEBUG: Task 614 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/72c14628-6091-442b-a066-3c5c6fb32876_calibrate_947422_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/72c14628-6091-442b-a066-3c5c6fb32876_calibrate_947422_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26e5d94a91ca24658c335548390f6a96 with result from task 200
parsl.dataflow.dflow DEBUG: Task 200 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79398958-a42e-436f-83e7-6f55265294e3_calibrate_947418_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/79398958-a42e-436f-83e7-6f55265294e3_calibrate_947418_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:42 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 1211 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b78edc394893ef5d3ee0d29817256a7 with result from task 1211
parsl.dataflow.dflow DEBUG: Task 1211 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/544db9e2-862d-4282-9b2b-ca9e9b5fba25_calibrate_947526_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 1211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/544db9e2-862d-4282-9b2b-ca9e9b5fba25_calibrate_947526_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 82de86439935e24db3d23b946de32095 with result from task 959
parsl.dataflow.dflow DEBUG: Task 959 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/85f1ddd5-07c0-4a7a-b281-ff68917d2781_calibrate_947524_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/85f1ddd5-07c0-4a7a-b281-ff68917d2781_calibrate_947524_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:43 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 714740b9097f2239217617b89b812644 with result from task 1028
parsl.dataflow.dflow DEBUG: Task 1028 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dd173ead-dcd3-4eca-8df4-b654e84fba0a_calibrate_947418_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/dd173ead-dcd3-4eca-8df4-b654e84fba0a_calibrate_947418_11.stderr
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a21e316185f702a3c9bd4bbff0108e65 with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d5808a35-8964-447b-8098-7b449935e370_calibrate_947528_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/d5808a35-8964-447b-8098-7b449935e370_calibrate_947528_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:44 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:45 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 1253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 94ca102ba7ce7b13a59400908ad89f10 with result from task 1253
parsl.dataflow.dflow DEBUG: Task 1253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b1e03d46-9207-4bcd-9018-258b5d115403_calibrate_947524_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 1253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b1e03d46-9207-4bcd-9018-258b5d115403_calibrate_947524_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:46 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 1121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07b5ba7872161bb0f53254c1c465f7da with result from task 1121
parsl.dataflow.dflow DEBUG: Task 1121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/ab7cd52d-a07d-46f5-b511-c938fe689ce2_calibrate_947526_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/ab7cd52d-a07d-46f5-b511-c938fe689ce2_calibrate_947526_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:47 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:48 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:49 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:50 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:51 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:30:52 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 14
parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 14
parsl.dataflow.strategy DEBUG: Executor multi has 14 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3e396af91964286a192316d1dc16287 with result from task 254
parsl.dataflow.dflow DEBUG: Task 254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/494efb28-18f9-4a1b-b438-0d0955748c08_calibrate_947418_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/494efb28-18f9-4a1b-b438-0d0955748c08_calibrate_947418_16.stderr
parsl.dataflow.dflow INFO: Task 1229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9b1295f628d917c999eb5923f813152d with result from task 1229
parsl.dataflow.dflow DEBUG: Task 1229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c2b6890a-9a2f-4618-8803-885abaa4ea43_calibrate_947528_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 1229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/c2b6890a-9a2f-4618-8803-885abaa4ea43_calibrate_947528_20.stderr
parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7677fd918ec71d9cb90d2b8d03ae24c7 with result from task 1006
parsl.dataflow.dflow DEBUG: Task 1006 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 1ae858c90c81b8dc696bc4c30e385d15
parsl.dataflow.memoization INFO: Task 1007 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8be0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 1299
parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2fb1adae-01ba-41f8-82f1-e4992beaa116_calibrate_947528_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2fb1adae-01ba-41f8-82f1-e4992beaa116_calibrate_947528_30.stderr
parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2943865f-60cf-4932-8399-28543b421fd8_characterizeImage_947528_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947528/2943865f-60cf-4932-8399-28543b421fd8_characterizeImage_947528_30.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 12
parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 542adbc2c4ed1574c08dc268dbae3ef1 with result from task 977
parsl.dataflow.dflow DEBUG: Task 977 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fd129370-5be8-44e4-9e51-9e34b8ea1993_calibrate_947418_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fd129370-5be8-44e4-9e51-9e34b8ea1993_calibrate_947418_24.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 11
parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 1207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cc8cbbb05a50213d45ed1e4fa6f22498 with result from task 1207
parsl.dataflow.dflow DEBUG: Task 1207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1208 has memoization hash 4cd98125ef2c08346381c3f7f53570ca
parsl.dataflow.memoization INFO: Task 1208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1208 try 0 launched on executor multi with executor id 1300
parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1d8e6da6-1ce3-4faf-8b49-2ccb126580ea_characterizeImage_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947524/1d8e6da6-1ce3-4faf-8b49-2ccb126580ea_characterizeImage_947524_61.stderr
parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27ac3833df71d89ff643c46cbe4f835a with result from task 1013
parsl.dataflow.dflow DEBUG: Task 1013 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f8f0f3c-c89f-4ee9-bdb9-248955d36b0f_calibrate_947528_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/8f8f0f3c-c89f-4ee9-bdb9-248955d36b0f_calibrate_947528_38.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 10
parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1192 try 1 failed
parsl.dataflow.dflow ERROR: Task 1192 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3583346baef57b8b83ea4e4b88958a with result from task 1192
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Task 1193 failed due to dependency failure
parsl.dataflow.dflow DEBUG: Task 1193 try 0 failed
parsl.dataflow.dflow INFO: Task 1193 failed due to dependency failure so skipping retries
parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/e6cdfd35-e482-46b1-b051-45ca6c1c0384_calibrate_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947420/e6cdfd35-e482-46b1-b051-45ca6c1c0384_calibrate_947420_61.stderr
parsl.dataflow.dflow INFO: Standard output for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947420/fad862df-9c9e-4ac2-8b1a-72413ef5ce4b_characterizeImage_947420_61.stderr
parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 461327b2213ef77c2e22f67a06a55f22 with result from task 992
parsl.dataflow.dflow DEBUG: Task 992 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b10a45fd-b7fa-4f8b-b977-c30b44f3c936_calibrate_947524_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/b10a45fd-b7fa-4f8b-b977-c30b44f3c936_calibrate_947524_25.stderr
2024-03-14 07:31:12 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:31:12 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 842 try 1 failed
parsl.dataflow.dflow ERROR: Task 842 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 3ce3e6645a807a73da36bb2cb694d99c with result from task 842
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/24262e80-a519-4047-962d-49edc776a01d_calibrate_947526_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:13 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:14 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:15 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 1145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77ca6fc496e346fd6486dafe552cc48a with result from task 1145
parsl.dataflow.dflow DEBUG: Task 1145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a89c029b-bb38-48f2-872d-534d2e6c07e2_calibrate_947528_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 1145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/a89c029b-bb38-48f2-872d-534d2e6c07e2_calibrate_947528_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:16 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:17 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:18 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:19 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:20 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:21 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:22 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:23 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:24 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:25 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:25 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:25 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:25 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:25 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 6
parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f41ed2b07e4b5a71ad978135435236a with result from task 194
parsl.dataflow.dflow DEBUG: Task 194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/75c3f1e8-b00b-42dc-b545-7108fdcf00f0_calibrate_947526_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947526/75c3f1e8-b00b-42dc-b545-7108fdcf00f0_calibrate_947526_30.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 5
parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:31:45 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:31:45 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:45 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:45 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:46 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:47 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:48 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ae858c90c81b8dc696bc4c30e385d15 with result from task 1007
parsl.dataflow.dflow DEBUG: Task 1007 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2fb1adae-01ba-41f8-82f1-e4992beaa116_calibrate_947528_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947528/2fb1adae-01ba-41f8-82f1-e4992beaa116_calibrate_947528_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:49 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:50 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:51 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:52 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:53 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:54 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:55 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1208 try 0 failed
parsl.dataflow.dflow INFO: Task 1208 marked for retry
parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stderr
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1208 has memoization hash 4cd98125ef2c08346381c3f7f53570ca
parsl.dataflow.memoization INFO: Task 1208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ac7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1208 try 1 launched on executor multi with executor id 1301
parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:56 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:57 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:58 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:31:58 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:18 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:19 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:20 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:21 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:22 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:23 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:24 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:25 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:26 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:27 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:28 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:29 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:30 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:31 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:31 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1db15c9b01152fcd8ac5235ea79aeef with result from task 901
parsl.dataflow.dflow DEBUG: Task 901 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 99222f61c1adcee189ba6a4f0aab3efc
parsl.dataflow.memoization INFO: Task 902 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8d4d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 1302
parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/fe33f3d8-5b79-4748-9f93-8c80f593be88_calibrate_947422_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/fe33f3d8-5b79-4748-9f93-8c80f593be88_calibrate_947422_58.stderr
parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e8570474-5be8-4913-919e-6ae08b807ab4_characterizeImage_947422_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947422/e8570474-5be8-4913-919e-6ae08b807ab4_characterizeImage_947422_58.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 4
parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.app.errors DEBUG: Reraising exception of type <class 'parsl.app.errors.BashExitFailure'>
parsl.dataflow.dflow DEBUG: Task 1208 try 1 failed
parsl.dataflow.dflow ERROR: Task 1208 failed after 1 retry attempts
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise
    reraise(t, v, v.__traceback__)
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper
    return func(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor
    raise pe.BashExitFailure(func_name, proc.returncode)
    ^^^^^^^^^^^^^^^^^
parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd98125ef2c08346381c3f7f53570ca with result from task 1208
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Standard output for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 1208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947524/559c78ef-ddb4-47c6-b09e-d92bc2c1c735_calibrate_947524_61.stderr
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:51 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:52 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:53 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:54 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:55 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:56 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:57 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:58 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:32:59 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:00 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:01 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:02 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:03 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:03 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:03 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:03 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:03 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:04 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:04 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ef87dfc8c5b4dc4d4de9b9a857bef11 with result from task 286
parsl.dataflow.dflow DEBUG: Task 286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 787ac16bc20b5a4a24ae208018221ae7
parsl.dataflow.memoization INFO: Task 287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8f01260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 1303
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0ef2d9aa-aa9f-4d6d-bc9b-eb420e7f7b87_calibrate_947418_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0ef2d9aa-aa9f-4d6d-bc9b-eb420e7f7b87_calibrate_947418_10.stderr
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e443c860-8a14-49ff-a2f3-912fa54126bc_characterizeImage_947418_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/e443c860-8a14-49ff-a2f3-912fa54126bc_characterizeImage_947418_10.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 3
parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99222f61c1adcee189ba6a4f0aab3efc with result from task 902
parsl.dataflow.dflow DEBUG: Task 902 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/fe33f3d8-5b79-4748-9f93-8c80f593be88_calibrate_947422_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947422/fe33f3d8-5b79-4748-9f93-8c80f593be88_calibrate_947422_58.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:33:24 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:33:24 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:24 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:24 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:24 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:25 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:26 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:27 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:28 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:29 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:30 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:31 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 787ac16bc20b5a4a24ae208018221ae7 with result from task 287
parsl.dataflow.dflow DEBUG: Task 287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0ef2d9aa-aa9f-4d6d-bc9b-eb420e7f7b87_calibrate_947418_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/0ef2d9aa-aa9f-4d6d-bc9b-eb420e7f7b87_calibrate_947418_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:32 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:33 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:34 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:34 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:35 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:36 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:33:37 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:58 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:33:59 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:00 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:01 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:02 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:03 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:04 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:05 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:05 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:05 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:06 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:07 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:08 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:09 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:10 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:11 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:11 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:31 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:32 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:33 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:34 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:35 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:35 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:35 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:35 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:35 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:35 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:36 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:36 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:36 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:36 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:37 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:38 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:39 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:40 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:41 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:42 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:43 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:44 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:44 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:44 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:44 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:44 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:34:44 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:35:04 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:35:04 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:04 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:05 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:06 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:06 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:06 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:06 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:07 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:08 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:09 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:10 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:11 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:12 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:13 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:14 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:15 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:16 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:17 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:18 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:18 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:18 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:18 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:18 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:18 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:35:38 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:35:38 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:38 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:39 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:40 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:41 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:42 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:43 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:44 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:45 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:46 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:47 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:48 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:49 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033355 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:50 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:51 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:51 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:35:51 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
17033356 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9dd2e2b64ee181c0f283e576102944e with result from task 229
parsl.dataflow.dflow DEBUG: Task 229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 230 has memoization hash ded51e849704ecb4be267f353e647765
parsl.dataflow.memoization INFO: Task 230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14a6f8ee16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 1304
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fda1c7a4-d974-4c99-ba5f-4fbadf448880_calibrate_947418_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fda1c7a4-d974-4c99-ba5f-4fbadf448880_calibrate_947418_1.stderr
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/69b9247c-afdc-4c08-bb30-1d9b8a367b77_characterizeImage_947418_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/characterizeImage/947418/69b9247c-afdc-4c08-bb30-1d9b8a367b77_characterizeImage_947418_1.stderr
2024-03-14 07:36:11 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:36:11 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:11 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:11 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:11 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:12 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:13 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:14 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:15 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:16 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:17 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:18 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:19 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:20 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:21 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:22 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:23 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:24 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:24 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 1
parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:44 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:45 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:46 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033299,17033300,17033301,17033302,17033303,17033304,17033305,17033306,17033307,17033308,17033309,17033310,17033311,17033312,17033313,17033314,17033315,17033316,17033317,17033318,17033319,17033320,17033321,17033322,17033323,17033324,17033325,17033326,17033327,17033328'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033325 R
17033326 R
17033327 R
17033328 R
17033315 R
17033316 R
17033317 R
17033318 R
17033319 R
17033320 R
17033321 R
17033322 R
17033323 R
17033324 R
17033312 R
17033313 R
17033314 R
17033304 R
17033305 R
17033306 R
17033307 R
17033308 R
17033309 R
17033310 R
17033311 R
17033300 R
17033301 R
17033302 R
17033303 R
17033299 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033325 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033326 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033327 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033328 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033315 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033316 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033317 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033318 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033319 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033320 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033321 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033322 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033323 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033324 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033312 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033313 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033314 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033304 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033305 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033306 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033307 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033308 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033309 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033310 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033311 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033300 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033301 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033302 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033303 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033299 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:47 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033329,17033330,17033331,17033332,17033333'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033329 R
17033330 R
17033331 R
17033332 R
17033333 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033329 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033330 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033331 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033332 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033333 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:48 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ded51e849704ecb4be267f353e647765 with result from task 230
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1280
parsl.dataflow.dflow INFO: Tasks in state States.failed: 12
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 10
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: Task 230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fda1c7a4-d974-4c99-ba5f-4fbadf448880_calibrate_947418_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/submit/DEEP/20201021/B1a/science#step1/20240314T072405Z/logs/calibrate/947418/fda1c7a4-d974-4c99-ba5f-4fbadf448880_calibrate_947418_1.stderr
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:49 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033334,17033335,17033336,17033337,17033338,17033339,17033340,17033341,17033342,17033343,17033344,17033345,17033346,17033347,17033348,17033349,17033350,17033351,17033352,17033353'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033353 R
17033347 R
17033348 R
17033349 R
17033350 R
17033351 R
17033352 R
17033342 R
17033343 R
17033344 R
17033345 R
17033346 R
17033334 R
17033335 R
17033336 R
17033337 R
17033338 R
17033339 R
17033340 R
17033341 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033353 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033347 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033348 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033349 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033350 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033351 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033352 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033342 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033343 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033344 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033345 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033346 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033334 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033335 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033336 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033337 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033338 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033339 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033340 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033341 with slurm status R to parsl state JobState.RUNNING
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:50 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:51 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:52 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:53 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:54 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 CG
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:55 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17033354,17033355,17033356,17033357,17033358,17033359,17033360,17033361,17033362,17033363,17033364,17033365,17033366,17033367,17033368,17033369,17033370,17033371,17033373,17033374,17033376,17033377,17033378,17033379,17033380,17033381,17033382,17033383,17033384,17033385,17033386,17033387,17033390,17033391,17033392,17033393,17033394,17033395,17033396,17033397,17033398,17033399,17033400,17033401,17033402,17033403,17033404,17033405,17033406,17033407,17033408,17033409'
parsl.providers.slurm.slurm DEBUG: squeue returned 17033356 CG
17033355 PD
17033401 R
17033402 R
17033403 R
17033404 R
17033405 R
17033406 R
17033407 R
17033408 R
17033409 R
17033398 R
17033399 R
17033400 R
17033397 R
17033393 R
17033394 R
17033395 R
17033396 R
17033387 R
17033390 R
17033391 R
17033392 R
17033377 R
17033378 R
17033379 R
17033380 R
17033381 R
17033382 R
17033383 R
17033384 R
17033385 R
17033386 R
17033376 R
17033367 R
17033368 R
17033369 R
17033370 R
17033371 R
17033373 R
17033374 R
17033357 R
17033358 R
17033359 R
17033360 R
17033361 R
17033362 R
17033363 R
17033364 R
17033365 R
17033366 R
17033354 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17033356 with slurm status CG to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033355 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17033401 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033402 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033403 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033404 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033405 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033406 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033407 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033408 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033409 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033398 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033399 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033400 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033397 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033393 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033394 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033395 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033396 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033387 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033390 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033391 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033392 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033377 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033378 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033379 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033380 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033381 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033382 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033383 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033384 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033385 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033386 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033376 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033367 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033368 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033369 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033370 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033371 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033373 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033374 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033357 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033358 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033359 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033360 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033361 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033362 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033363 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033364 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033365 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033366 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17033354 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating missing job 17033372 to completed status
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 108, active_tasks = 0
parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 108/0 running/pending blocks, and 473 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks
parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (108) than minimum blocks (0)
parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in
parsl.dataflow.strategy DEBUG: Idle time 2.86102294921875e-06s is less than max_idletime 120.0s for executor multi; not scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22707413500688
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=109, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 108 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01ebd2bae4d2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01ebd2bae4d2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: caf185e3c0b0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: caf185e3c0b0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8dd3c2c92462
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8dd3c2c92462
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d625e0c7fb0a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d625e0c7fb0a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43f126743443
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43f126743443
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be70b7ceb45f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be70b7ceb45f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d34a1065c645
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d34a1065c645
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d11426873e7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d11426873e7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 50a082bc6b5e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 50a082bc6b5e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be44d5a61d64
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be44d5a61d64
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 23d263b17707
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 23d263b17707
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78f05bb0d63a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78f05bb0d63a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60b34100fe57
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60b34100fe57
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8c8c65d48d3b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8c8c65d48d3b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7eed8344522f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7eed8344522f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67b614e6bb23
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67b614e6bb23
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e13040f98b3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e13040f98b3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df15cedf5dbe
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df15cedf5dbe
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d574b99a42a5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d574b99a42a5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 275b9fcf30ca
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 275b9fcf30ca
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 351b684f6a49
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 351b684f6a49
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64715518b3e8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64715518b3e8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64a85ed61612
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64a85ed61612
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 817e46a671bb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 817e46a671bb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ac1cf0653cd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ac1cf0653cd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f578e658d75d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f578e658d75d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3af59bd83863
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3af59bd83863
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c03da78b4aa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c03da78b4aa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0092fddc045
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0092fddc045
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c76d688c1baa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c76d688c1baa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9d26bb2cd36
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9d26bb2cd36
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 103653c3078a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 103653c3078a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 96761f7047c5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 96761f7047c5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d34b204ddb18
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d34b204ddb18
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ca4cd18c18dd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ca4cd18c18dd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: de3123c8b74f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: de3123c8b74f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ebbf8fcadd9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ebbf8fcadd9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9a5a1520b766
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9a5a1520b766
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4fed101b22c4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4fed101b22c4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b5f66f47e5d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b5f66f47e5d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9cda05ea61b9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9cda05ea61b9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3516f1176763
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3516f1176763
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 135454302bad
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 135454302bad
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e1c8e58f7274
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e1c8e58f7274
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b0534a28e34
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b0534a28e34
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 125ea16161cb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 125ea16161cb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f846b322576
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f846b322576
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 71d168d4ff9f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 71d168d4ff9f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bd941306b390
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bd941306b390
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97c3fd4d8af0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97c3fd4d8af0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c5cd70c2a96c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c5cd70c2a96c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce10b9136565
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce10b9136565
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f07813ceedc1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f07813ceedc1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 12895cc89ca1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 12895cc89ca1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 229e1b13f5e7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 229e1b13f5e7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2fd1c53e9870
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2fd1c53e9870
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d0ac50389d9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d0ac50389d9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 65b03702b9dc
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 65b03702b9dc
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d7b998ae937
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d7b998ae937
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 805896e961a9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 805896e961a9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9e7f5e57b2f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9e7f5e57b2f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bd1704deb7a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bd1704deb7a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b83ebadfd61
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b83ebadfd61
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c18b21290563
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c18b21290563
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 22fb1585b996
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 22fb1585b996
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e9a8c03c380
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e9a8c03c380
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9982d49b2562
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9982d49b2562
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b472805f0f0d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b472805f0f0d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25ccb8b5476a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25ccb8b5476a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fb343171a5c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fb343171a5c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf7107cec274
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf7107cec274
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af804f1108d0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af804f1108d0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec8daa9d874e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec8daa9d874e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 92b84949d23e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 92b84949d23e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0b797ffff780
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0b797ffff780
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 659dbb53e0f3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 659dbb53e0f3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66c4f41439b1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66c4f41439b1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9b3b95d8033c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9b3b95d8033c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ca878fb3794
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ca878fb3794
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8e55f7816dcd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8e55f7816dcd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c1d5f2bc55e4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c1d5f2bc55e4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e94576cccd63
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e94576cccd63
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 710b24032add
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 710b24032add
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2f67329e673
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2f67329e673
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7c3ca32bf5b9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7c3ca32bf5b9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 461788a27b9a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 461788a27b9a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9137838349a0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9137838349a0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07303caf9812
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07303caf9812
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 21529a90305e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 21529a90305e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d266616ee9a5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d266616ee9a5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd5d96529e00
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd5d96529e00
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d492f24ec77c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d492f24ec77c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4b8b42fcb6b5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4b8b42fcb6b5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b4943c86f4a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b4943c86f4a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69608c7f62c1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69608c7f62c1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec9d72e42626
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec9d72e42626
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9849bac34122
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9849bac34122
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a02f8a39e6ee
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a02f8a39e6ee
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b46c830cc956
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b46c830cc956
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0356d135f47f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0356d135f47f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2c45067716e8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2c45067716e8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1643a4605135
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1643a4605135
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 32ceb2a49b0c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 32ceb2a49b0c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4fd26fa2d044
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4fd26fa2d044
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01f9fbf5a0e0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01f9fbf5a0e0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 477e8494b2fe
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 477e8494b2fe
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2bc8be1e57e6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2bc8be1e57e6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a60d93ffbfa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a60d93ffbfa
2024-03-14 07:36:56 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
2024-03-14 07:36:56 proc_lsst.multi:201 [INFO]  cancelling 17033331 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033331 on provider escience_rtx6k
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033405 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033405 on provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033344 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033344 on provider escience_a40
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033370 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033370 on provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033310 on provider astro
proc_lsst.multi INFO: cancelling 17033310 on provider astro
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033377 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033377 on provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033354 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033354 on provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033304 on provider astro
proc_lsst.multi INFO: cancelling 17033304 on provider astro
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033330 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033330 on provider escience_rtx6k
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033384 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033384 on provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033363 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033363 on provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
2024-03-14 07:36:57 proc_lsst.multi:201 [INFO]  cancelling 17033397 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033397 on provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033348 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033348 on provider escience_a40
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033392 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033392 on provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033322 on provider astro
proc_lsst.multi INFO: cancelling 17033322 on provider astro
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033385 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033385 on provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033358 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033358 on provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033406 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033406 on provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033303 on provider astro
proc_lsst.multi INFO: cancelling 17033303 on provider astro
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033335 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033335 on provider escience_a40
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033327 on provider astro
proc_lsst.multi INFO: cancelling 17033327 on provider astro
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 14686 on provider local
proc_lsst.multi INFO: cancelling 14686 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 14686
2024-03-14 07:36:58 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
2024-03-14 07:36:58 proc_lsst.multi:201 [INFO]  cancelling 17033390 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033390 on provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033378 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033378 on provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033313 on provider astro
proc_lsst.multi INFO: cancelling 17033313 on provider astro
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033391 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033391 on provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033367 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033367 on provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033350 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033350 on provider escience_a40
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033409 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033409 on provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033338 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033338 on provider escience_a40
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033393 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033393 on provider checkpoint
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033308 on provider astro
proc_lsst.multi INFO: cancelling 17033308 on provider astro
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033326 on provider astro
proc_lsst.multi INFO: cancelling 17033326 on provider astro
2024-03-14 07:36:59 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
2024-03-14 07:36:59 proc_lsst.multi:201 [INFO]  cancelling 17033333 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033333 on provider escience_rtx6k
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033361 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033361 on provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033362 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033362 on provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033369 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033369 on provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033299 on provider astro
proc_lsst.multi INFO: cancelling 17033299 on provider astro
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033404 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033404 on provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033374 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033374 on provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033356 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033356 on provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033351 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033351 on provider escience_a40
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033325 on provider astro
proc_lsst.multi INFO: cancelling 17033325 on provider astro
2024-03-14 07:37:00 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
2024-03-14 07:37:00 proc_lsst.multi:201 [INFO]  cancelling 17033400 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033400 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033387 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033387 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033395 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033395 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033368 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033368 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033386 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033386 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033408 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033408 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033382 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033382 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033355 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033355 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033399 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033399 on provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033315 on provider astro
proc_lsst.multi INFO: cancelling 17033315 on provider astro
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033309 on provider astro
proc_lsst.multi INFO: cancelling 17033309 on provider astro
2024-03-14 07:37:01 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
2024-03-14 07:37:01 proc_lsst.multi:201 [INFO]  cancelling 17033401 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033401 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033301 on provider astro
proc_lsst.multi INFO: cancelling 17033301 on provider astro
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033359 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033359 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033380 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033380 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033398 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033398 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033403 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033403 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033373 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033373 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033407 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033407 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033402 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033402 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033381 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033381 on provider checkpoint
2024-03-14 07:37:02 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
2024-03-14 07:37:02 proc_lsst.multi:201 [INFO]  cancelling 17033316 on provider astro
proc_lsst.multi INFO: cancelling 17033316 on provider astro
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033321 on provider astro
proc_lsst.multi INFO: cancelling 17033321 on provider astro
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033328 on provider astro
proc_lsst.multi INFO: cancelling 17033328 on provider astro
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033352 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033352 on provider escience_a40
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033360 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033360 on provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033319 on provider astro
proc_lsst.multi INFO: cancelling 17033319 on provider astro
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033349 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033349 on provider escience_a40
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033317 on provider astro
proc_lsst.multi INFO: cancelling 17033317 on provider astro
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033364 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033364 on provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033371 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033371 on provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033306 on provider astro
proc_lsst.multi INFO: cancelling 17033306 on provider astro
2024-03-14 07:37:03 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
2024-03-14 07:37:03 proc_lsst.multi:201 [INFO]  cancelling 17033376 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033376 on provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033365 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033365 on provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033379 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033379 on provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033343 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033343 on provider escience_a40
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033383 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033383 on provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033323 on provider astro
proc_lsst.multi INFO: cancelling 17033323 on provider astro
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033353 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033353 on provider escience_a40
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033346 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033346 on provider escience_a40
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033312 on provider astro
proc_lsst.multi INFO: cancelling 17033312 on provider astro
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033396 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033396 on provider checkpoint
2024-03-14 07:37:04 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
2024-03-14 07:37:04 proc_lsst.multi:201 [INFO]  cancelling 17033314 on provider astro
proc_lsst.multi INFO: cancelling 17033314 on provider astro
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033307 on provider astro
proc_lsst.multi INFO: cancelling 17033307 on provider astro
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033357 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033357 on provider checkpoint
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033337 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033337 on provider escience_a40
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033320 on provider astro
proc_lsst.multi INFO: cancelling 17033320 on provider astro
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033302 on provider astro
proc_lsst.multi INFO: cancelling 17033302 on provider astro
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033305 on provider astro
proc_lsst.multi INFO: cancelling 17033305 on provider astro
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033334 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033334 on provider escience_a40
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033394 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033394 on provider checkpoint
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033332 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033332 on provider escience_rtx6k
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033329 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033329 on provider escience_rtx6k
2024-03-14 07:37:05 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
2024-03-14 07:37:05 proc_lsst.multi:201 [INFO]  cancelling 17033366 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033366 on provider checkpoint
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033311 on provider astro
proc_lsst.multi INFO: cancelling 17033311 on provider astro
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033336 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033336 on provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033347 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033347 on provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033342 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033342 on provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033340 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033340 on provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033339 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033339 on provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033324 on provider astro
proc_lsst.multi INFO: cancelling 17033324 on provider astro
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033345 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033345 on provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033318 on provider astro
proc_lsst.multi INFO: cancelling 17033318 on provider astro
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033300 on provider astro
proc_lsst.multi INFO: cancelling 17033300 on provider astro
2024-03-14 07:37:06 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
2024-03-14 07:37:06 proc_lsst.multi:201 [INFO]  cancelling 17033341 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033341 on provider escience_a40
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-14 07:37:07 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-14 07:37:07 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['17033354', '17033355', '17033356', '17033357', '17033358', '17033359', '17033360', '17033361', '17033362', '17033363', '17033364', '17033365', '17033366', '17033367', '17033368', '17033369', '17033370', '17033371', '17033372', '17033373', '17033374', '17033376', '17033377', '17033378', '17033379', '17033380', '17033381', '17033382', '17033383', '17033384', '17033385', '17033386', '17033387', '17033390', '17033391', '17033392', '17033393', '17033394', '17033395', '17033396', '17033397', '17033398', '17033399', '17033400', '17033401', '17033402', '17033403', '17033404', '17033405', '17033406', '17033407', '17033408', '17033409', '17033299', '17033300', '17033301', '17033302', '17033303', '17033304', '17033305', '17033306', '17033307', '17033308', '17033309', '17033310', '17033311', '17033312', '17033313', '17033314', '17033315', '17033316', '17033317', '17033318', '17033319', '17033320', '17033321', '17033322', '17033323', '17033324', '17033325', '17033326', '17033327', '17033328', '17033334', '17033335', '17033336', '17033337', '17033338', '17033339', '17033340', '17033341', '17033342', '17033343', '17033344', '17033345', '17033346', '17033347', '17033348', '17033349', '17033350', '17033351', '17033352', '17033353', '17033329', '17033330', '17033331', '17033332', '17033333', '14686']
proc_lsst.multi INFO: new jobs since last cancel ['17033354', '17033355', '17033356', '17033357', '17033358', '17033359', '17033360', '17033361', '17033362', '17033363', '17033364', '17033365', '17033366', '17033367', '17033368', '17033369', '17033370', '17033371', '17033372', '17033373', '17033374', '17033376', '17033377', '17033378', '17033379', '17033380', '17033381', '17033382', '17033383', '17033384', '17033385', '17033386', '17033387', '17033390', '17033391', '17033392', '17033393', '17033394', '17033395', '17033396', '17033397', '17033398', '17033399', '17033400', '17033401', '17033402', '17033403', '17033404', '17033405', '17033406', '17033407', '17033408', '17033409', '17033299', '17033300', '17033301', '17033302', '17033303', '17033304', '17033305', '17033306', '17033307', '17033308', '17033309', '17033310', '17033311', '17033312', '17033313', '17033314', '17033315', '17033316', '17033317', '17033318', '17033319', '17033320', '17033321', '17033322', '17033323', '17033324', '17033325', '17033326', '17033327', '17033328', '17033334', '17033335', '17033336', '17033337', '17033338', '17033339', '17033340', '17033341', '17033342', '17033343', '17033344', '17033345', '17033346', '17033347', '17033348', '17033349', '17033350', '17033351', '17033352', '17033353', '17033329', '17033330', '17033331', '17033332', '17033333', '14686']
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033354 in provider checkpoint
proc_lsst.multi INFO: found job 17033354 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033354 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033354 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033355 in provider checkpoint
proc_lsst.multi INFO: found job 17033355 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033355 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033355 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033356 in provider checkpoint
proc_lsst.multi INFO: found job 17033356 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033356 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033356 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033357 in provider checkpoint
proc_lsst.multi INFO: found job 17033357 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033357 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033357 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033358 in provider checkpoint
proc_lsst.multi INFO: found job 17033358 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033358 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033358 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033359 in provider checkpoint
proc_lsst.multi INFO: found job 17033359 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033359 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033359 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033360 in provider checkpoint
proc_lsst.multi INFO: found job 17033360 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033360 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033360 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033361 in provider checkpoint
proc_lsst.multi INFO: found job 17033361 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033361 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033361 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033362 in provider checkpoint
proc_lsst.multi INFO: found job 17033362 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033362 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033362 on provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:146 [INFO]  found job 17033363 in provider checkpoint
proc_lsst.multi INFO: found job 17033363 in provider checkpoint
2024-03-14 07:37:07 proc_lsst.multi:201 [INFO]  cancelling 17033363 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033363 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033364 in provider checkpoint
proc_lsst.multi INFO: found job 17033364 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033364 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033364 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033365 in provider checkpoint
proc_lsst.multi INFO: found job 17033365 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033365 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033365 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033366 in provider checkpoint
proc_lsst.multi INFO: found job 17033366 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033366 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033366 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033367 in provider checkpoint
proc_lsst.multi INFO: found job 17033367 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033367 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033367 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033368 in provider checkpoint
proc_lsst.multi INFO: found job 17033368 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033368 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033368 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033369 in provider checkpoint
proc_lsst.multi INFO: found job 17033369 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033369 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033369 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033370 in provider checkpoint
proc_lsst.multi INFO: found job 17033370 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033370 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033370 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033371 in provider checkpoint
proc_lsst.multi INFO: found job 17033371 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033371 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033371 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033372 in provider checkpoint
proc_lsst.multi INFO: found job 17033372 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033372 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033372 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033373 in provider checkpoint
proc_lsst.multi INFO: found job 17033373 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033373 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033373 on provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:146 [INFO]  found job 17033374 in provider checkpoint
proc_lsst.multi INFO: found job 17033374 in provider checkpoint
2024-03-14 07:37:08 proc_lsst.multi:201 [INFO]  cancelling 17033374 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033374 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033376 in provider checkpoint
proc_lsst.multi INFO: found job 17033376 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033376 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033376 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033377 in provider checkpoint
proc_lsst.multi INFO: found job 17033377 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033377 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033377 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033378 in provider checkpoint
proc_lsst.multi INFO: found job 17033378 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033378 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033378 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033379 in provider checkpoint
proc_lsst.multi INFO: found job 17033379 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033379 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033379 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033380 in provider checkpoint
proc_lsst.multi INFO: found job 17033380 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033380 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033380 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033381 in provider checkpoint
proc_lsst.multi INFO: found job 17033381 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033381 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033381 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033382 in provider checkpoint
proc_lsst.multi INFO: found job 17033382 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033382 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033382 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033383 in provider checkpoint
proc_lsst.multi INFO: found job 17033383 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033383 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033383 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033384 in provider checkpoint
proc_lsst.multi INFO: found job 17033384 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033384 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033384 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033385 in provider checkpoint
proc_lsst.multi INFO: found job 17033385 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033385 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033385 on provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:146 [INFO]  found job 17033386 in provider checkpoint
proc_lsst.multi INFO: found job 17033386 in provider checkpoint
2024-03-14 07:37:09 proc_lsst.multi:201 [INFO]  cancelling 17033386 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033386 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033387 in provider checkpoint
proc_lsst.multi INFO: found job 17033387 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033387 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033387 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033390 in provider checkpoint
proc_lsst.multi INFO: found job 17033390 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033390 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033390 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033391 in provider checkpoint
proc_lsst.multi INFO: found job 17033391 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033391 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033391 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033392 in provider checkpoint
proc_lsst.multi INFO: found job 17033392 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033392 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033392 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033393 in provider checkpoint
proc_lsst.multi INFO: found job 17033393 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033393 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033393 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033394 in provider checkpoint
proc_lsst.multi INFO: found job 17033394 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033394 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033394 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033395 in provider checkpoint
proc_lsst.multi INFO: found job 17033395 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033395 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033395 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033396 in provider checkpoint
proc_lsst.multi INFO: found job 17033396 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033396 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033396 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033397 in provider checkpoint
proc_lsst.multi INFO: found job 17033397 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033397 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033397 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033398 in provider checkpoint
proc_lsst.multi INFO: found job 17033398 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033398 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033398 on provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:146 [INFO]  found job 17033399 in provider checkpoint
proc_lsst.multi INFO: found job 17033399 in provider checkpoint
2024-03-14 07:37:10 proc_lsst.multi:201 [INFO]  cancelling 17033399 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033399 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033400 in provider checkpoint
proc_lsst.multi INFO: found job 17033400 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033400 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033400 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033401 in provider checkpoint
proc_lsst.multi INFO: found job 17033401 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033401 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033401 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033402 in provider checkpoint
proc_lsst.multi INFO: found job 17033402 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033402 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033402 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033403 in provider checkpoint
proc_lsst.multi INFO: found job 17033403 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033403 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033403 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033404 in provider checkpoint
proc_lsst.multi INFO: found job 17033404 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033404 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033404 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033405 in provider checkpoint
proc_lsst.multi INFO: found job 17033405 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033405 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033405 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033406 in provider checkpoint
proc_lsst.multi INFO: found job 17033406 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033406 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033406 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033407 in provider checkpoint
proc_lsst.multi INFO: found job 17033407 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033407 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033407 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033408 in provider checkpoint
proc_lsst.multi INFO: found job 17033408 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033408 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033408 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033409 in provider checkpoint
proc_lsst.multi INFO: found job 17033409 in provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033409 on provider checkpoint
proc_lsst.multi INFO: cancelling 17033409 on provider checkpoint
2024-03-14 07:37:11 proc_lsst.multi:146 [INFO]  found job 17033299 in provider astro
proc_lsst.multi INFO: found job 17033299 in provider astro
2024-03-14 07:37:11 proc_lsst.multi:201 [INFO]  cancelling 17033299 on provider astro
proc_lsst.multi INFO: cancelling 17033299 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033300 in provider astro
proc_lsst.multi INFO: found job 17033300 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033300 on provider astro
proc_lsst.multi INFO: cancelling 17033300 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033301 in provider astro
proc_lsst.multi INFO: found job 17033301 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033301 on provider astro
proc_lsst.multi INFO: cancelling 17033301 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033302 in provider astro
proc_lsst.multi INFO: found job 17033302 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033302 on provider astro
proc_lsst.multi INFO: cancelling 17033302 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033303 in provider astro
proc_lsst.multi INFO: found job 17033303 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033303 on provider astro
proc_lsst.multi INFO: cancelling 17033303 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033304 in provider astro
proc_lsst.multi INFO: found job 17033304 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033304 on provider astro
proc_lsst.multi INFO: cancelling 17033304 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033305 in provider astro
proc_lsst.multi INFO: found job 17033305 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033305 on provider astro
proc_lsst.multi INFO: cancelling 17033305 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033306 in provider astro
proc_lsst.multi INFO: found job 17033306 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033306 on provider astro
proc_lsst.multi INFO: cancelling 17033306 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033307 in provider astro
proc_lsst.multi INFO: found job 17033307 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033307 on provider astro
proc_lsst.multi INFO: cancelling 17033307 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033308 in provider astro
proc_lsst.multi INFO: found job 17033308 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033308 on provider astro
proc_lsst.multi INFO: cancelling 17033308 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033309 in provider astro
proc_lsst.multi INFO: found job 17033309 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033309 on provider astro
proc_lsst.multi INFO: cancelling 17033309 on provider astro
2024-03-14 07:37:12 proc_lsst.multi:146 [INFO]  found job 17033310 in provider astro
proc_lsst.multi INFO: found job 17033310 in provider astro
2024-03-14 07:37:12 proc_lsst.multi:201 [INFO]  cancelling 17033310 on provider astro
proc_lsst.multi INFO: cancelling 17033310 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033311 in provider astro
proc_lsst.multi INFO: found job 17033311 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033311 on provider astro
proc_lsst.multi INFO: cancelling 17033311 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033312 in provider astro
proc_lsst.multi INFO: found job 17033312 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033312 on provider astro
proc_lsst.multi INFO: cancelling 17033312 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033313 in provider astro
proc_lsst.multi INFO: found job 17033313 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033313 on provider astro
proc_lsst.multi INFO: cancelling 17033313 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033314 in provider astro
proc_lsst.multi INFO: found job 17033314 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033314 on provider astro
proc_lsst.multi INFO: cancelling 17033314 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033315 in provider astro
proc_lsst.multi INFO: found job 17033315 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033315 on provider astro
proc_lsst.multi INFO: cancelling 17033315 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033316 in provider astro
proc_lsst.multi INFO: found job 17033316 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033316 on provider astro
proc_lsst.multi INFO: cancelling 17033316 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033317 in provider astro
proc_lsst.multi INFO: found job 17033317 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033317 on provider astro
proc_lsst.multi INFO: cancelling 17033317 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033318 in provider astro
proc_lsst.multi INFO: found job 17033318 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033318 on provider astro
proc_lsst.multi INFO: cancelling 17033318 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033319 in provider astro
proc_lsst.multi INFO: found job 17033319 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033319 on provider astro
proc_lsst.multi INFO: cancelling 17033319 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033320 in provider astro
proc_lsst.multi INFO: found job 17033320 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033320 on provider astro
proc_lsst.multi INFO: cancelling 17033320 on provider astro
2024-03-14 07:37:13 proc_lsst.multi:146 [INFO]  found job 17033321 in provider astro
proc_lsst.multi INFO: found job 17033321 in provider astro
2024-03-14 07:37:13 proc_lsst.multi:201 [INFO]  cancelling 17033321 on provider astro
proc_lsst.multi INFO: cancelling 17033321 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033322 in provider astro
proc_lsst.multi INFO: found job 17033322 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033322 on provider astro
proc_lsst.multi INFO: cancelling 17033322 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033323 in provider astro
proc_lsst.multi INFO: found job 17033323 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033323 on provider astro
proc_lsst.multi INFO: cancelling 17033323 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033324 in provider astro
proc_lsst.multi INFO: found job 17033324 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033324 on provider astro
proc_lsst.multi INFO: cancelling 17033324 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033325 in provider astro
proc_lsst.multi INFO: found job 17033325 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033325 on provider astro
proc_lsst.multi INFO: cancelling 17033325 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033326 in provider astro
proc_lsst.multi INFO: found job 17033326 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033326 on provider astro
proc_lsst.multi INFO: cancelling 17033326 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033327 in provider astro
proc_lsst.multi INFO: found job 17033327 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033327 on provider astro
proc_lsst.multi INFO: cancelling 17033327 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033328 in provider astro
proc_lsst.multi INFO: found job 17033328 in provider astro
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033328 on provider astro
proc_lsst.multi INFO: cancelling 17033328 on provider astro
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033334 in provider escience_a40
proc_lsst.multi INFO: found job 17033334 in provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033334 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033334 on provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033335 in provider escience_a40
proc_lsst.multi INFO: found job 17033335 in provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033335 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033335 on provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033336 in provider escience_a40
proc_lsst.multi INFO: found job 17033336 in provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033336 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033336 on provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:146 [INFO]  found job 17033337 in provider escience_a40
proc_lsst.multi INFO: found job 17033337 in provider escience_a40
2024-03-14 07:37:14 proc_lsst.multi:201 [INFO]  cancelling 17033337 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033337 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033338 in provider escience_a40
proc_lsst.multi INFO: found job 17033338 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033338 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033338 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033339 in provider escience_a40
proc_lsst.multi INFO: found job 17033339 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033339 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033339 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033340 in provider escience_a40
proc_lsst.multi INFO: found job 17033340 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033340 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033340 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033341 in provider escience_a40
proc_lsst.multi INFO: found job 17033341 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033341 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033341 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033342 in provider escience_a40
proc_lsst.multi INFO: found job 17033342 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033342 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033342 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033343 in provider escience_a40
proc_lsst.multi INFO: found job 17033343 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033343 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033343 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033344 in provider escience_a40
proc_lsst.multi INFO: found job 17033344 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033344 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033344 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033345 in provider escience_a40
proc_lsst.multi INFO: found job 17033345 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033345 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033345 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033346 in provider escience_a40
proc_lsst.multi INFO: found job 17033346 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033346 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033346 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033347 in provider escience_a40
proc_lsst.multi INFO: found job 17033347 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033347 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033347 on provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:146 [INFO]  found job 17033348 in provider escience_a40
proc_lsst.multi INFO: found job 17033348 in provider escience_a40
2024-03-14 07:37:15 proc_lsst.multi:201 [INFO]  cancelling 17033348 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033348 on provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033349 in provider escience_a40
proc_lsst.multi INFO: found job 17033349 in provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033349 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033349 on provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033350 in provider escience_a40
proc_lsst.multi INFO: found job 17033350 in provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033350 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033350 on provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033351 in provider escience_a40
proc_lsst.multi INFO: found job 17033351 in provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033351 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033351 on provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033352 in provider escience_a40
proc_lsst.multi INFO: found job 17033352 in provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033352 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033352 on provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033353 in provider escience_a40
proc_lsst.multi INFO: found job 17033353 in provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033353 on provider escience_a40
proc_lsst.multi INFO: cancelling 17033353 on provider escience_a40
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033329 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033329 in provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033329 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033329 on provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033330 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033330 in provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033330 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033330 on provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033331 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033331 in provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033331 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033331 on provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033332 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033332 in provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033332 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033332 on provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 17033333 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17033333 in provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 17033333 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17033333 on provider escience_rtx6k
2024-03-14 07:37:16 proc_lsst.multi:146 [INFO]  found job 14686 in provider local
proc_lsst.multi INFO: found job 14686 in provider local
2024-03-14 07:37:16 proc_lsst.multi:201 [INFO]  cancelling 14686 on provider local
proc_lsst.multi INFO: cancelling 14686 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 14686
parsl.providers.local.local WARNING: Failed to kill PID: 14686 and child processes on local
2024-03-14 07:37:18 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 751.0899 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201021_B1a_science#step1_20240314T072405Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 751.0961 seconds; current memory usage: 0.401 Gibyte, delta: 0.012 Gibyte, peak delta: 0.012 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 784.9750 seconds; current memory usage: 0.401 Gibyte, delta: 0.264 Gibyte, peak delta: 0.264 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.401 Gibyte (main), 0.401 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20201021_B1a_science#step1_20240314T072405Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	13m7.460s
user	3m18.048s
sys	1m58.541s